[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:26 dist-test-slave-21h8 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:26 dist-test-slave-21h8 krb5kdc[399](info): set up 2 sockets
Sep 26 09:32:26 dist-test-slave-21h8 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 09:32:27.147843 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:27.274684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41979
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:27.359781 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:27.360126 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:27.360167 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:27.361418 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:27.361490 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:27.361516 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:27.361547 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:27.361577 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:27.363114 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:27.363425 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:27.363698 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:27.369825 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:27.369819 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:27.369885 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:27.369995 412 server_base.cc:1034] running on GCE node
I20260926 09:32:27.370276 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:27.370599 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:27.371775 412 hybrid_clock.cc:648] HybridClock initialized: now 1790415147371750 us; error 40 us; skew 500 ppm
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:27.379622 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:27.380111 412 webserver.cc:460] Webserver started at http://127.0.98.62:42267/ using document root <none> and password file <none>
I20260926 09:32:27.380318 412 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:27.380362 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:27.380457 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:27.381551 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "8b384c057f5a4699a91816c93790f792"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c1f527c2ce8268dc4429debabed21fd1"
server_key_iv: "fc4ba03b36fd93902816d8296625d55a"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.381872 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b384c057f5a4699a91816c93790f792"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c1f527c2ce8268dc4429debabed21fd1"
server_key_iv: "fc4ba03b36fd93902816d8296625d55a"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.383277 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:27.384238 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.384536 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:27.384768 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "8b384c057f5a4699a91816c93790f792"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c1f527c2ce8268dc4429debabed21fd1"
server_key_iv: "fc4ba03b36fd93902816d8296625d55a"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.384851 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:27.406726 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:27.408972 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:27.409148 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:27.413268 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41979
I20260926 09:32:27.413326 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41979 every 8 connection(s)
I20260926 09:32:27.413734 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:27.414669 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:27.417035 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792: Bootstrap starting.
I20260926 09:32:27.417583 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:27.417816 480 log.cc:824] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:27.418481 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792: No bootstrap required, opened a new log
I20260926 09:32:27.420128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 412
I20260926 09:32:27.420091 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b384c057f5a4699a91816c93790f792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41979 } }
I20260926 09:32:27.420217 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:27.420250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:27.420266 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b384c057f5a4699a91816c93790f792, State: Initialized, Role: FOLLOWER
I20260926 09:32:27.420428 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b384c057f5a4699a91816c93790f792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41979 } }
I20260926 09:32:27.420491 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:27.420547 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:27.420595 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:27.420984 392 external_mini_cluster.cc:1428] Setting key ebdf0de8e4a842f66e03f49094f835fb
I20260926 09:32:27.421176 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b384c057f5a4699a91816c93790f792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41979 } }
I20260926 09:32:27.421294 480 leader_election.cc:302] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b384c057f5a4699a91816c93790f792; no voters:
I20260926 09:32:27.421466 480 leader_election.cc:288] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:27.421522 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:27.421645 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [term 1 LEADER]: Becoming Leader. State: Replica: 8b384c057f5a4699a91816c93790f792, State: Running, Role: LEADER
I20260926 09:32:27.421762 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:27.421737 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b384c057f5a4699a91816c93790f792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41979 } }
I20260926 09:32:27.422538 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b384c057f5a4699a91816c93790f792. Latest consensus state: current_term: 1 leader_uuid: "8b384c057f5a4699a91816c93790f792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b384c057f5a4699a91816c93790f792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41979 } } }
I20260926 09:32:27.422643 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:27.422804 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b384c057f5a4699a91816c93790f792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b384c057f5a4699a91816c93790f792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41979 } } }
I20260926 09:32:27.422843 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:27.422873 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:27.423458 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:27.425319 494 catalog_manager.cc:1347] Generated new cluster ID: 19652bece3f742f59c3cbc3fddb2948a
I20260926 09:32:27.425377 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:27.432102 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:27.432654 494 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:27.438244 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b384c057f5a4699a91816c93790f792: Generated new TSK 0
I20260926 09:32:27.438445 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:32:27.440580 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.422386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.0.98.62:41979)
0926 09:32:27.422587 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:27.422596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:27.423205 (+ 609us) server_negotiation.cc:407] Connection header received
0926 09:32:27.425740 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:27.425748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:27.425805 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:27.425976 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:27.427960 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.429447 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.431027 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.431747 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.431873 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.436588 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:27.436594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:27.436595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:27.436608 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:27.438631 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.439061 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.439064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.439065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.439176 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.439499 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.439500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.439500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.439784 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:27.439926 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:27.440167 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 09:32:27.440339 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:32:27.469913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41979
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:27.553104 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:27.553308 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:27.553330 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:27.554636 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:27.554680 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:27.554708 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:27.556258 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:27.556525 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:27.556787 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:27.561910 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:27.561947 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:27.562002 506 server_base.cc:1034] running on GCE node
W20260926 09:32:27.561920 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:27.562302 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:27.562551 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:27.563701 506 hybrid_clock.cc:648] HybridClock initialized: now 1790415147563681 us; error 39 us; skew 500 ppm
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:27.566123 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:27.566605 506 webserver.cc:460] Webserver started at http://127.0.98.1:45475/ using document root <none> and password file <none>
I20260926 09:32:27.566833 506 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:27.566879 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:27.567034 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:27.568157 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "2552c98c6e9b4ca3ad8e846f4df23b0a"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c6e753ea36214ebc501e3442e34a7eb6"
server_key_iv: "a2ba8a6164a79699e9e115fab9eac4d4"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.568524 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2552c98c6e9b4ca3ad8e846f4df23b0a"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c6e753ea36214ebc501e3442e34a7eb6"
server_key_iv: "a2ba8a6164a79699e9e115fab9eac4d4"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.569854 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:32:27.570564 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.570749 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:32:27.570839 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "2552c98c6e9b4ca3ad8e846f4df23b0a"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c6e753ea36214ebc501e3442e34a7eb6"
server_key_iv: "a2ba8a6164a79699e9e115fab9eac4d4"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.570911 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:27.584494 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:27.586630 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:27.586809 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:27.587076 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:27.587457 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:27.587514 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.587556 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:27.587591 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.594030 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43523
I20260926 09:32:27.594097 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43523 every 8 connection(s)
I20260926 09:32:27.594419 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:32:27.594794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 506
I20260926 09:32:27.594890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:27.595078 392 external_mini_cluster.cc:1428] Setting key eccd79c01c0b64967a341e68c960549c
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:32:27.605656 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.595171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57321 (local address 127.0.98.62:41979)
0926 09:32:27.595230 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:27.595232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:27.595547 (+ 315us) server_negotiation.cc:407] Connection header received
0926 09:32:27.596227 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:27.596228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:27.596237 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:27.596287 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:27.597651 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.598438 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.599614 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.599949 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.600032 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.602585 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:27.602586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:27.602587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:27.602596 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:27.604308 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.604731 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.604732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.604732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.604787 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.605008 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.605008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.605008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.605115 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:27.605209 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:27.605502 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 09:32:27.605585 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:32:27.605885 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.595136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41979 (local address 127.0.98.1:57321)
0926 09:32:27.595350 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:27.595403 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:27.596124 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:27.596383 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:27.596388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:27.597260 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:27.597550 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:27.597556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.598549 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.598550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:27.599521 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:27.599524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.600062 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.600063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:27.600178 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.600815 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:27.600824 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:27.602474 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:27.604414 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:27.604420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:27.604422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:27.604652 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:27.604849 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:27.604849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:27.604849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:27.604951 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:27.605212 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:27.605218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:27.605380 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 09:32:27.605473 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 09:32:27.605623 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260926 09:32:27.606575 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41979
I20260926 09:32:27.606699 635 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 09:32:27.606923 635 heartbeater.cc:511] Master 127.0.98.62:41979 requested a full tablet report, sending...
I20260926 09:32:27.607573 444 ts_manager.cc:194] Registered new tserver with Master: 2552c98c6e9b4ca3ad8e846f4df23b0a (127.0.98.1:43523)
I20260926 09:32:27.608227 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:27.622865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41979
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:27.707214 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:27.707466 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:27.707492 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:27.708617 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:27.708653 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:27.708683 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:27.710059 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:27.710249 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:27.710480 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:27.715773 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:27.715796 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:27.715786 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:27.716089 640 server_base.cc:1034] running on GCE node
I20260926 09:32:27.716308 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:27.716503 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:27.717667 640 hybrid_clock.cc:648] HybridClock initialized: now 1790415147717652 us; error 32 us; skew 500 ppm
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:27.720180 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:27.720613 640 webserver.cc:460] Webserver started at http://127.0.98.2:38887/ using document root <none> and password file <none>
I20260926 09:32:27.720849 640 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:27.720916 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:27.721074 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:27.722294 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "97cb5e1a4b534c50bcef2b0d5098f2b8"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "5978d617a5a199f5a80a0e668a08f2b3"
server_key_iv: "8b98ce6c37ebc7ce88146ba16c4fe896"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.722666 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97cb5e1a4b534c50bcef2b0d5098f2b8"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "5978d617a5a199f5a80a0e668a08f2b3"
server_key_iv: "8b98ce6c37ebc7ce88146ba16c4fe896"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.724125 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 09:32:27.724788 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.724960 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:27.725071 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "97cb5e1a4b534c50bcef2b0d5098f2b8"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "5978d617a5a199f5a80a0e668a08f2b3"
server_key_iv: "8b98ce6c37ebc7ce88146ba16c4fe896"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.725144 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:27.752737 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:27.755280 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:27.755489 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:27.755743 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:27.756062 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:27.756116 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.756157 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:27.756203 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.762586 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45589
I20260926 09:32:27.762631 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45589 every 8 connection(s)
I20260926 09:32:27.762953 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:27.768110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 640
I20260926 09:32:27.768214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:27.768386 392 external_mini_cluster.cc:1428] Setting key 7352fc3d8f8bb3df8220244ca022d899
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:27.773736 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.763745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56057 (local address 127.0.98.62:41979)
0926 09:32:27.763836 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:27.763839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:27.764040 (+ 201us) server_negotiation.cc:407] Connection header received
0926 09:32:27.764604 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:27.764606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:27.764619 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:27.764676 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:27.765691 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.766189 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.767220 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.767472 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.767558 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.770227 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:27.770229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:27.770229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:27.770237 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:27.772168 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.772668 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.772668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.772669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.772718 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.773028 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.773028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.773028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.773131 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:27.773182 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:27.773545 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 09:32:27.773669 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:32:27.773983 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.763731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41979 (local address 127.0.98.2:56057)
0926 09:32:27.763900 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:27.763946 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:27.764466 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:27.764767 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:27.764777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:27.765363 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:27.765563 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:27.765570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.766282 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.766283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:27.767104 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:27.767107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.767568 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.767569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:27.767709 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.768215 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:27.768226 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:27.769792 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:27.772256 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:27.772262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:27.772264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:27.772591 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:27.772785 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:27.772786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:27.772786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:27.772956 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:27.773199 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:27.773205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:27.773385 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 09:32:27.773515 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 09:32:27.773727 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260926 09:32:27.774505 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41979
I20260926 09:32:27.774643 769 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:27.774839 769 heartbeater.cc:511] Master 127.0.98.62:41979 requested a full tablet report, sending...
I20260926 09:32:27.775233 444 ts_manager.cc:194] Registered new tserver with Master: 97cb5e1a4b534c50bcef2b0d5098f2b8 (127.0.98.2:45589)
I20260926 09:32:27.775628 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56057
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:27.795017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41979
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:27.880153 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:27.880419 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:27.880465 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:27.881726 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:27.881801 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:27.881850 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:27.883524 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:27.883814 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:27.884091 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:32:27.889488 774 server_base.cc:1034] running on GCE node
W20260926 09:32:27.889470 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:27.889470 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:27.889470 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:27.889973 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:27.890277 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:27.891441 774 hybrid_clock.cc:648] HybridClock initialized: now 1790415147891430 us; error 46 us; skew 500 ppm
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:27.893893 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:27.894364 774 webserver.cc:460] Webserver started at http://127.0.98.3:44745/ using document root <none> and password file <none>
I20260926 09:32:27.894611 774 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:27.894683 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:27.894834 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:27.896013 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ffb2eca0c1540b586dd3c048e173228"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c702d3eb5cc8ba07eb935fbff60122cb"
server_key_iv: "a3bfd2160c86ae8829fd93da23ae25be"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.896389 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ffb2eca0c1540b586dd3c048e173228"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c702d3eb5cc8ba07eb935fbff60122cb"
server_key_iv: "a3bfd2160c86ae8829fd93da23ae25be"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.897776 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:27.898478 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.898659 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:27.898762 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "3ffb2eca0c1540b586dd3c048e173228"
format_stamp: "Formatted at 2026-09-26 09:32:27 on dist-test-slave-21h8"
server_key: "c702d3eb5cc8ba07eb935fbff60122cb"
server_key_iv: "a3bfd2160c86ae8829fd93da23ae25be"
server_key_version: "encryptionkey@0"
I20260926 09:32:27.898845 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:27.912848 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:27.915508 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:27.915678 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:27.915935 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:27.916270 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:27.916324 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.916368 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:27.916405 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:27.924047 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41297
I20260926 09:32:27.924120 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41297 every 8 connection(s)
I20260926 09:32:27.924503 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:32:27.930018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 774
I20260926 09:32:27.930136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:27.930325 392 external_mini_cluster.cc:1428] Setting key ed28f9c176e2902dc1b97595dc2b08e1
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:27.934600 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.925254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58341 (local address 127.0.98.62:41979)
0926 09:32:27.925319 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:27.925321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:27.925548 (+ 227us) server_negotiation.cc:407] Connection header received
0926 09:32:27.926051 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:27.926052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:27.926064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:27.926125 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:27.927124 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.927706 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.928886 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.929174 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.929237 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.931946 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:27.931947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:27.931947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:27.931954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:27.933311 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.933722 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.933723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.933723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.933777 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.934064 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.934065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.934065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.934171 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:27.934221 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:27.934464 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 09:32:27.934529 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:32:27.934827 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.925258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41979 (local address 127.0.98.3:58341)
0926 09:32:27.925404 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:27.925442 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:27.925929 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:27.926195 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:27.926198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:27.926815 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:27.927018 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:27.927022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.927787 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.927788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:27.928784 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:27.928786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.929251 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.929252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:27.929358 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.929950 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:27.929959 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:27.931862 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:27.933391 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:27.933398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:27.933400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:27.933651 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:27.933875 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:27.933876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:27.933876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:27.933997 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:27.934227 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:27.934231 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:27.934350 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 09:32:27.934437 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 09:32:27.934565 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260926 09:32:27.935465 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41979
I20260926 09:32:27.935562 903 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:27.935719 903 heartbeater.cc:511] Master 127.0.98.62:41979 requested a full tablet report, sending...
I20260926 09:32:27.936072 444 ts_manager.cc:194] Registered new tserver with Master: 3ffb2eca0c1540b586dd3c048e173228 (127.0.98.3:41297)
I20260926 09:32:27.936414 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58341
I20260926 09:32:27.942412 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:27.965528 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.957957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.0.98.1:43523)
0926 09:32:27.958135 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:27.958140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:27.958151 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:32:27.958179 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:27.958181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:27.958195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:27.958273 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:27.958832 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.959640 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.960357 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.960681 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.960736 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.962987 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:27.962997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:27.962997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:27.963010 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:27.964530 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.964865 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.964868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.964869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.964923 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.965085 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.965086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.965086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.965215 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:27.965317 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:27.965382 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 09:32:27.965443 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260926 09:32:27.965955 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43282
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:32:27 dist-test-slave-21h8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:27.989156 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:27.981569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.0.98.1:43523)
0926 09:32:27.981642 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:27.981645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:27.981760 (+ 115us) server_negotiation.cc:407] Connection header received
0926 09:32:27.981862 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:27.981863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:27.981883 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:27.981948 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:27.982537 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.983036 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.983891 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:27.984101 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:27.984177 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:27.986535 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:27.986537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:27.986537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:27.986545 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:27.987978 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.988303 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.988304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.988304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.988354 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:27.988565 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:27.988566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:27.988566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:27.988664 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:27.988759 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:27.989018 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 09:32:27.989087 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:32:27.990134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 506
I20260926 09:32:27.995085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 640
I20260926 09:32:28.000939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 774
I20260926 09:32:28.005224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 412
2026-09-26T09:32:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1901 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:28 dist-test-slave-21h8 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:28 dist-test-slave-21h8 krb5kdc[917](info): set up 2 sockets
Sep 26 09:32:28 dist-test-slave-21h8 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 09:32:29.042925 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:29.163800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:29.245229 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:29.245498 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:29.245538 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:29.246749 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:29.246826 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:29.246850 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:29.246869 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:29.246901 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:29.249163 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:29.249478 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:29.249733 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:29.254791 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:29.254724 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:29.254724 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:29.254985 930 server_base.cc:1034] running on GCE node
I20260926 09:32:29.255307 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:29.255698 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:29.256865 930 hybrid_clock.cc:648] HybridClock initialized: now 1790415149256841 us; error 42 us; skew 500 ppm
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:29.259613 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:29.260110 930 webserver.cc:460] Webserver started at http://127.0.98.62:37393/ using document root <none> and password file <none>
I20260926 09:32:29.260368 930 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:29.260414 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:29.260592 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:29.261709 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "5770ec62ac484c8483ffcfb91984dba0"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "71e9d4f68f823f8f9e8eea628698fbf1"
server_key_iv: "a21409e2132724e2be97578db08a5d99"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.262084 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "5770ec62ac484c8483ffcfb91984dba0"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "71e9d4f68f823f8f9e8eea628698fbf1"
server_key_iv: "a21409e2132724e2be97578db08a5d99"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.263625 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 09:32:29.264424 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.264599 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:29.264700 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "5770ec62ac484c8483ffcfb91984dba0"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "71e9d4f68f823f8f9e8eea628698fbf1"
server_key_iv: "a21409e2132724e2be97578db08a5d99"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.264773 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:29.283505 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:29.285616 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:29.285769 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:29.289963 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37157
I20260926 09:32:29.290000 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37157 every 8 connection(s)
I20260926 09:32:29.290382 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:29.291460 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:29.293843 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0: Bootstrap starting.
I20260926 09:32:29.294454 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:29.294763 998 log.cc:824] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:29.295567 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0: No bootstrap required, opened a new log
I20260926 09:32:29.297169 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5770ec62ac484c8483ffcfb91984dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37157 } }
I20260926 09:32:29.297318 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:29.297343 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5770ec62ac484c8483ffcfb91984dba0, State: Initialized, Role: FOLLOWER
I20260926 09:32:29.297526 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5770ec62ac484c8483ffcfb91984dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37157 } }
I20260926 09:32:29.297593 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:29.297616 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:29.297685 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:29.298216 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5770ec62ac484c8483ffcfb91984dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37157 } }
I20260926 09:32:29.298334 998 leader_election.cc:302] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5770ec62ac484c8483ffcfb91984dba0; no voters:
I20260926 09:32:29.298554 998 leader_election.cc:288] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:29.298585 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:29.298622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 930
I20260926 09:32:29.298705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:29.298861 392 external_mini_cluster.cc:1428] Setting key 5bc3fedca5a815a5b4a4c048acb2d1db
I20260926 09:32:29.298857 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:29.298915 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [term 1 LEADER]: Becoming Leader. State: Replica: 5770ec62ac484c8483ffcfb91984dba0, State: Running, Role: LEADER
I20260926 09:32:29.299002 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5770ec62ac484c8483ffcfb91984dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37157 } }
I20260926 09:32:29.299351 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5770ec62ac484c8483ffcfb91984dba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5770ec62ac484c8483ffcfb91984dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37157 } } }
I20260926 09:32:29.299454 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:29.299678 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:29.299707 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5770ec62ac484c8483ffcfb91984dba0. Latest consensus state: current_term: 1 leader_uuid: "5770ec62ac484c8483ffcfb91984dba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5770ec62ac484c8483ffcfb91984dba0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37157 } } }
I20260926 09:32:29.299865 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:29.300218 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:29.302011 1012 catalog_manager.cc:1347] Generated new cluster ID: 77f5bfe8786c4605afb6bc2471116f9e
I20260926 09:32:29.302078 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:29.311793 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.299483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.0.98.62:37157)
0926 09:32:29.300039 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:29.300050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:29.300293 (+ 243us) server_negotiation.cc:407] Connection header received
0926 09:32:29.300416 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:29.300424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:29.300576 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:29.300722 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:29.301549 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.302938 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.304104 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.304760 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.304876 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.307849 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:29.307855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:29.307855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:29.307869 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:29.309954 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.310399 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.310403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.310403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.310531 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.310755 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.310756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.310757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.311081 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:29.311247 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:29.311390 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 09:32:29.311558 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":77,"threads_started":1}
I20260926 09:32:29.312233 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:29.312860 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:29.317469 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5770ec62ac484c8483ffcfb91984dba0: Generated new TSK 0
I20260926 09:32:29.317684 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:32:29.347355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37157
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:29.431778 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:29.432054 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:29.432082 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:29.433318 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:29.433364 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:29.433393 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:29.434931 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:29.435217 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:29.435491 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:29.440542 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:29.440603 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:29.440552 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:29.440835 1024 server_base.cc:1034] running on GCE node
I20260926 09:32:29.441071 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:29.441327 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:29.442494 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790415149442477 us; error 27 us; skew 500 ppm
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:29.445060 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:29.445525 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46641/ using document root <none> and password file <none>
I20260926 09:32:29.445772 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:29.445843 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:29.445999 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:29.447098 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "87f7e6a3e9ce4539a39f39e1a116ba76"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "c35d948aa07a38c171cc81ee516af179"
server_key_iv: "8ec17239ef52fc612226fe4eb6a6e569"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.447502 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "87f7e6a3e9ce4539a39f39e1a116ba76"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "c35d948aa07a38c171cc81ee516af179"
server_key_iv: "8ec17239ef52fc612226fe4eb6a6e569"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.448854 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:29.449678 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.449872 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:32:29.449995 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "87f7e6a3e9ce4539a39f39e1a116ba76"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "c35d948aa07a38c171cc81ee516af179"
server_key_iv: "8ec17239ef52fc612226fe4eb6a6e569"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.450073 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:29.472339 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:29.474346 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:29.474530 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:29.474784 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:29.475117 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:29.475175 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.475210 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:29.475260 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.481122 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40547
I20260926 09:32:29.481170 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40547 every 8 connection(s)
I20260926 09:32:29.481496 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:32:29.482652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 1024
I20260926 09:32:29.482743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:29.482911 392 external_mini_cluster.cc:1428] Setting key e977bea08a5012eb5be6abc47b40db53
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:32:29.492671 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.482246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53467 (local address 127.0.98.62:37157)
0926 09:32:29.482328 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:29.482331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:29.482532 (+ 201us) server_negotiation.cc:407] Connection header received
0926 09:32:29.483386 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:29.483388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:29.483401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:29.483475 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:29.484490 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.485165 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.486412 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.486782 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.486869 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.489399 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:29.489400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:29.489401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:29.489411 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:29.491208 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.491741 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.491742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.491743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.491808 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.492066 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.492067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.492067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.492181 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:29.492236 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:29.492480 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 09:32:29.492602 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:32:29.492952 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.482203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37157 (local address 127.0.98.1:53467)
0926 09:32:29.482385 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:29.482422 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:29.483150 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:29.483586 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:29.483591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:29.484181 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:29.484378 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.484383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.485274 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.485275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.486311 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.486314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.486894 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.486895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.487036 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.487655 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:29.487663 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:29.489296 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:29.491360 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:29.491371 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:29.491378 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:29.491678 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:29.491883 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:29.491884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:29.491884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:29.491993 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:29.492240 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:29.492244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:29.492367 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 09:32:29.492452 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:32:29.492696 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260926 09:32:29.493521 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37157
I20260926 09:32:29.493644 1153 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:29.493839 1153 heartbeater.cc:511] Master 127.0.98.62:37157 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 09:32:29.494423 962 ts_manager.cc:194] Registered new tserver with Master: 87f7e6a3e9ce4539a39f39e1a116ba76 (127.0.98.1:40547)
I20260926 09:32:29.495395 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:29.510303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37157
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:29.592621 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:29.592880 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:29.592921 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:29.594128 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:29.594210 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:29.594259 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:29.595851 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:29.596136 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:29.596374 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:29.601980 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:29.601979 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:29.601974 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:29.602252 1158 server_base.cc:1034] running on GCE node
I20260926 09:32:29.602437 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:29.602696 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:29.603895 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790415149603882 us; error 46 us; skew 500 ppm
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:29.606673 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:29.607152 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43657/ using document root <none> and password file <none>
I20260926 09:32:29.607470 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:29.607540 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:29.607703 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:29.608822 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "885f207e917042ff9d0e5f53807a0ba6"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "8b843658272fcbec4999ba55feb53729"
server_key_iv: "fb8d09786f188487e5f4f89f17cb53bf"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.609207 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "885f207e917042ff9d0e5f53807a0ba6"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "8b843658272fcbec4999ba55feb53729"
server_key_iv: "fb8d09786f188487e5f4f89f17cb53bf"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.610592 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:29.611405 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.611611 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:29.611711 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "885f207e917042ff9d0e5f53807a0ba6"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "8b843658272fcbec4999ba55feb53729"
server_key_iv: "fb8d09786f188487e5f4f89f17cb53bf"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.611799 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:29.622586 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:29.625058 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:29.625216 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:29.625453 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:29.625761 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:29.625823 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.625866 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:29.625895 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.631603 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36089
I20260926 09:32:29.631657 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36089 every 8 connection(s)
I20260926 09:32:29.631981 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:29.635144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 1158
I20260926 09:32:29.635254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:29.635452 392 external_mini_cluster.cc:1428] Setting key a1ae1c720d05e1c663b3907fd49f1d03
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:29.642189 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.632691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44449 (local address 127.0.98.62:37157)
0926 09:32:29.632777 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:29.632780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:29.632973 (+ 193us) server_negotiation.cc:407] Connection header received
0926 09:32:29.633545 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:29.633547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:29.633558 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:29.633613 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:29.634564 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.635079 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.636247 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.636474 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.636564 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.639189 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:29.639191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:29.639191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:29.639199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:29.640728 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.641169 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.641171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.641171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.641236 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.641504 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.641513 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.641513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.641639 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:29.641718 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:29.642001 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 09:32:29.642091 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:32:29.642444 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.632657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37157 (local address 127.0.98.2:44449)
0926 09:32:29.632836 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:29.632883 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:29.633437 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:29.633697 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:29.633701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:29.634268 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:29.634466 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.634470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.635174 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.635175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.636152 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.636154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.636574 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.636575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.636712 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.637393 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:29.637406 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:29.639081 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:29.640836 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:29.640843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:29.640846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:29.641088 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:29.641304 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:29.641305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:29.641305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:29.641422 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:29.641709 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:29.641716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:29.641867 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 09:32:29.641981 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:32:29.642158 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260926 09:32:29.642894 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37157
I20260926 09:32:29.642992 1287 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:32:29.643231 1287 heartbeater.cc:511] Master 127.0.98.62:37157 requested a full tablet report, sending...
I20260926 09:32:29.643694 962 ts_manager.cc:194] Registered new tserver with Master: 885f207e917042ff9d0e5f53807a0ba6 (127.0.98.2:36089)
I20260926 09:32:29.644043 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:29.661846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37157
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:29.744155 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:29.744385 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:29.744411 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:29.745579 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:29.745620 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:29.745647 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:29.747087 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37157
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:29.747296 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:29.747574 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:29.752730 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:29.752785 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:29.752772 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:29.753000 1292 server_base.cc:1034] running on GCE node
I20260926 09:32:29.753252 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:29.753566 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:29.754738 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790415149754717 us; error 37 us; skew 500 ppm
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:29.757418 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:29.757889 1292 webserver.cc:460] Webserver started at http://127.0.98.3:41947/ using document root <none> and password file <none>
I20260926 09:32:29.758121 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:29.758189 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:29.758347 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:29.759505 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "2f99111328ca4d8bb3767978cdfe6528"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "3c1d1e0b8f24b1e2ebad7c98ecd4cb05"
server_key_iv: "4bde5a04f7f722d66a766dabbe7a8df8"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.759871 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2f99111328ca4d8bb3767978cdfe6528"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "3c1d1e0b8f24b1e2ebad7c98ecd4cb05"
server_key_iv: "4bde5a04f7f722d66a766dabbe7a8df8"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.761229 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:32:29.762084 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.762259 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:29.762398 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "2f99111328ca4d8bb3767978cdfe6528"
format_stamp: "Formatted at 2026-09-26 09:32:29 on dist-test-slave-21h8"
server_key: "3c1d1e0b8f24b1e2ebad7c98ecd4cb05"
server_key_iv: "4bde5a04f7f722d66a766dabbe7a8df8"
server_key_version: "encryptionkey@0"
I20260926 09:32:29.762486 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:29.776940 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:29.779450 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:29.779618 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:29.779868 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:29.780193 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:29.780249 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.780292 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:29.780347 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:29.786192 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34001
I20260926 09:32:29.786234 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34001 every 8 connection(s)
I20260926 09:32:29.786608 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:32:29 dist-test-slave-21h8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:29.796756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 1292
I20260926 09:32:29.796856 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.787411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52433 (local address 127.0.98.62:37157)
0926 09:32:29.787483 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:29.787485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:29.787721 (+ 236us) server_negotiation.cc:407] Connection header received
0926 09:32:29.788262 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:29.788263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:29.788274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:29.788330 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:29.789377 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.789898 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.791055 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.791271 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.791459 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.793673 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:29.793674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:29.793674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:29.793682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:29.795378 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.795783 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.795784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.795784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.795847 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.796090 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.796091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.796091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.796228 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:29.796284 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:29.796622 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 09:32:29.796724 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:32:29.796866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:29.797048 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.787413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37157 (local address 127.0.98.3:52433)
0926 09:32:29.787602 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:29.787656 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:29.788139 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:29.788399 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:29.788403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:29.789013 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:29.789221 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.789226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.789993 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.789994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.790945 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.790948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.791472 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.791473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.791595 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.792133 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:29.792142 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:29.793585 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:29.795466 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:29.795472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:29.795474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:29.795687 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:29.795919 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:29.795920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:29.795920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:29.796025 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:29.796318 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:29.796322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:29.796452 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 09:32:29.796565 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 09:32:29.796701 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260926 09:32:29.797060 392 external_mini_cluster.cc:1428] Setting key 16373421a50e9bc8c18756b2c6fee12f
I20260926 09:32:29.797710 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37157
I20260926 09:32:29.797842 1421 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:29.798071 1421 heartbeater.cc:511] Master 127.0.98.62:37157 requested a full tablet report, sending...
I20260926 09:32:29.798412 961 ts_manager.cc:194] Registered new tserver with Master: 2f99111328ca4d8bb3767978cdfe6528 (127.0.98.3:34001)
I20260926 09:32:29.798868 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52433
I20260926 09:32:29.809497 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:32:29.818073 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.811242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.0.98.62:37157)
0926 09:32:29.811364 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:29.811367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:29.811461 (+ 94us) server_negotiation.cc:407] Connection header received
0926 09:32:29.811504 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:29.811505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:29.811515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:29.811571 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:29.812332 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.812917 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.813716 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.813915 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.813990 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.815465 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:29.815466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:29.815467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:29.815474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:29.816789 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.817172 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.817173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.817174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.817230 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:29.817433 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:29.817434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:29.817435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:29.817649 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:29.817729 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:29.817834 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 09:32:29.817929 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:32:29.820570 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:29.822243 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:29.828826 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.824547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36089 (local address 127.0.0.1:51206)
0926 09:32:29.824655 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:29.824698 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:29.824763 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:29.825088 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:29.825092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:29.825094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:29.825173 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.825175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.826365 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.826367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.827600 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.827603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.828440 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.828441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.828543 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.828564 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 09:32:29.828684 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 09:32:29.828744 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":60,"threads_started":1}
I20260926 09:32:29.828835 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.824660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.0.98.2:36089)
0926 09:32:29.824818 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:29.824824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:29.824837 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:32:29.824880 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:29.824882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:29.824899 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:29.824992 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:29.825258 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.826250 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.827699 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.828338 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.828438 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.828506 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:29.828704 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 09:32:29.828745 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260926 09:32:29.829000 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.824920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40547 (local address 127.0.0.1:51128)
0926 09:32:29.825122 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:29.825131 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:29.825167 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:29.825441 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:29.825441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:29.825443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:29.825474 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.825475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.826865 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.826866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.827826 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.827828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.828763 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.828764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.828830 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.828834 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:29.828864 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 09:32:29.828912 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260926 09:32:29.829013 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.824754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34001 (local address 127.0.0.1:53214)
0926 09:32:29.824974 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:29.824985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:29.825030 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:29.825324 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:29.825325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:29.825327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:29.825397 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.825399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.826792 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.826793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.827929 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:29.827931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.828812 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.828812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:29.828867 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.828870 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:29.828897 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 09:32:29.828925 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":33,"thread_start_us":83,"threads_started":1}
I20260926 09:32:29.829617 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.825132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.0.98.1:40547)
0926 09:32:29.825296 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:29.825301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:29.825310 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:29.825346 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:29.825348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:29.825362 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:29.825432 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:29.825568 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.826733 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.827916 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.828654 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.829384 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.829453 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:29.829480 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 09:32:29.829536 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260926 09:32:29.829675 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0054685ceb451691ea9d49793a38e1 (DEFAULT_TABLE table=test-table [id=ca2976e904f04abaafcfa8125343673f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:29.830101 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0054685ceb451691ea9d49793a38e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:29.830256 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0054685ceb451691ea9d49793a38e1 (DEFAULT_TABLE table=test-table [id=ca2976e904f04abaafcfa8125343673f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:29.830574 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0054685ceb451691ea9d49793a38e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:29.833702 1439 tablet_bootstrap.cc:492] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76: Bootstrap starting.
I20260926 09:32:29.833892 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:29.824927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.0.98.3:34001)
0926 09:32:29.825133 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:29.825138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:29.825150 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:32:29.825190 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:29.825192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:29.825209 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:29.825319 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:29.825573 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.826672 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.827996 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:29.828728 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:29.829117 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:29.833725 (+ 4608us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:29.833757 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 09:32:29.833810 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260926 09:32:29.833921 1438 tablet_bootstrap.cc:492] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6: Bootstrap starting.
I20260926 09:32:29.834457 1439 tablet_bootstrap.cc:654] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:29.834503 1438 tablet_bootstrap.cc:654] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:29.834564 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0054685ceb451691ea9d49793a38e1 (DEFAULT_TABLE table=test-table [id=ca2976e904f04abaafcfa8125343673f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:29.834828 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0054685ceb451691ea9d49793a38e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:29.835070 1439 log.cc:824] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:29.835064 1438 log.cc:824] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:29.835857 1439 tablet_bootstrap.cc:492] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76: No bootstrap required, opened a new log
I20260926 09:32:29.835932 1439 ts_tablet_manager.cc:1397] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:29.835938 1438 tablet_bootstrap.cc:492] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6: No bootstrap required, opened a new log
I20260926 09:32:29.836015 1438 ts_tablet_manager.cc:1397] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:29.837468 1438 raft_consensus.cc:348] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:29.837638 1438 raft_consensus.cc:374] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:29.837682 1438 raft_consensus.cc:729] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885f207e917042ff9d0e5f53807a0ba6, State: Initialized, Role: FOLLOWER
I20260926 09:32:29.837800 1438 consensus_queue.cc:260] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:29.838119 1438 ts_tablet_manager.cc:1428] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:29.838231 1287 heartbeater.cc:503] Master 127.0.98.62:37157 was elected leader, sending a full tablet report...
I20260926 09:32:29.838299 1442 tablet_bootstrap.cc:492] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528: Bootstrap starting.
I20260926 09:32:29.838271 1439 raft_consensus.cc:348] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:29.838433 1439 raft_consensus.cc:374] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:29.838495 1439 raft_consensus.cc:729] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f7e6a3e9ce4539a39f39e1a116ba76, State: Initialized, Role: FOLLOWER
I20260926 09:32:29.838630 1439 consensus_queue.cc:260] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:29.838917 1153 heartbeater.cc:503] Master 127.0.98.62:37157 was elected leader, sending a full tablet report...
I20260926 09:32:29.838997 1442 tablet_bootstrap.cc:654] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:29.838907 1439 ts_tablet_manager.cc:1428] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:32:29.839757 1442 log.cc:824] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:29.840476 1442 tablet_bootstrap.cc:492] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528: No bootstrap required, opened a new log
I20260926 09:32:29.840562 1442 ts_tablet_manager.cc:1397] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:29.841980 1442 raft_consensus.cc:348] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:29.842160 1442 raft_consensus.cc:374] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:29.842202 1442 raft_consensus.cc:729] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f99111328ca4d8bb3767978cdfe6528, State: Initialized, Role: FOLLOWER
I20260926 09:32:29.842324 1442 consensus_queue.cc:260] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:29.842630 1442 ts_tablet_manager.cc:1428] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:29.842643 1421 heartbeater.cc:503] Master 127.0.98.62:37157 was elected leader, sending a full tablet report...
W20260926 09:32:29.882634 1288 tablet.cc:2369] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:32:29.982259 1154 tablet.cc:2369] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:32:30.037196 1422 tablet.cc:2369] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:32:30.182972 1443 raft_consensus.cc:482] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:30.183132 1443 raft_consensus.cc:504] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:30.183542 1443 leader_election.cc:288] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f99111328ca4d8bb3767978cdfe6528 (127.0.98.3:34001), 87f7e6a3e9ce4539a39f39e1a116ba76 (127.0.98.1:40547)
I20260926 09:32:30.186717 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:30.184137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47015 (local address 127.0.98.1:40547)
0926 09:32:30.184214 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:30.184217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:30.184264 (+ 47us) server_negotiation.cc:407] Connection header received
0926 09:32:30.184321 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:30.184321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:30.184337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:30.184390 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:30.184561 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.185026 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.185976 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.186356 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.186410 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:30.186450 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:30.186599 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 09:32:30.186640 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 09:32:30.186753 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:30.183885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35577 (local address 127.0.98.3:34001)
0926 09:32:30.183973 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:30.183976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:30.184037 (+ 61us) server_negotiation.cc:407] Connection header received
0926 09:32:30.184090 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:30.184091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:30.184103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:30.184182 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:30.184398 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.184960 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.185982 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.186363 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.186452 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:30.186461 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:30.186643 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 09:32:30.186689 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 09:32:30.186733 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:30.184025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40547 (local address 127.0.98.2:47015)
0926 09:32:30.184201 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:30.184213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:30.184275 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:30.184408 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:30.184409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:30.184411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:30.184474 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:30.184476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.185109 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.185110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:30.185877 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:30.185879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.186442 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.186444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:30.186524 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:30.186527 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:30.186571 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:32:30.186629 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":3,"thread_start_us":56,"threads_started":1}
I20260926 09:32:30.186741 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:30.183809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34001 (local address 127.0.98.2:35577)
0926 09:32:30.183960 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:30.183973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:30.184032 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:30.184207 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:30.184208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:30.184210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:30.184290 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:30.184293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.185048 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.185049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:30.185876 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:30.185878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.186475 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.186476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:30.186541 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:30.186545 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:30.186619 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:32:30.186656 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260926 09:32:30.187163 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0054685ceb451691ea9d49793a38e1" candidate_uuid: "885f207e917042ff9d0e5f53807a0ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" is_pre_election: true
I20260926 09:32:30.187397 1107 raft_consensus.cc:2454] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 885f207e917042ff9d0e5f53807a0ba6 in term 0.
I20260926 09:32:30.187410 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0054685ceb451691ea9d49793a38e1" candidate_uuid: "885f207e917042ff9d0e5f53807a0ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f99111328ca4d8bb3767978cdfe6528" is_pre_election: true
I20260926 09:32:30.187585 1375 raft_consensus.cc:2454] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 885f207e917042ff9d0e5f53807a0ba6 in term 0.
I20260926 09:32:30.187649 1177 leader_election.cc:302] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87f7e6a3e9ce4539a39f39e1a116ba76, 885f207e917042ff9d0e5f53807a0ba6; no voters:
I20260926 09:32:30.187779 1443 raft_consensus.cc:2781] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:30.187857 1443 raft_consensus.cc:482] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:30.187876 1443 raft_consensus.cc:3037] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:30.189034 1443 raft_consensus.cc:504] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:30.189178 1443 leader_election.cc:288] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [CANDIDATE]: Term 1 election: Requested vote from peers 2f99111328ca4d8bb3767978cdfe6528 (127.0.98.3:34001), 87f7e6a3e9ce4539a39f39e1a116ba76 (127.0.98.1:40547)
I20260926 09:32:30.189373 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0054685ceb451691ea9d49793a38e1" candidate_uuid: "885f207e917042ff9d0e5f53807a0ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f99111328ca4d8bb3767978cdfe6528"
I20260926 09:32:30.189389 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0054685ceb451691ea9d49793a38e1" candidate_uuid: "885f207e917042ff9d0e5f53807a0ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76"
I20260926 09:32:30.189459 1375 raft_consensus.cc:3037] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:30.189479 1107 raft_consensus.cc:3037] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:30.190203 1375 raft_consensus.cc:2454] T 3a0054685ceb451691ea9d49793a38e1 P 2f99111328ca4d8bb3767978cdfe6528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885f207e917042ff9d0e5f53807a0ba6 in term 1.
I20260926 09:32:30.190207 1107 raft_consensus.cc:2454] T 3a0054685ceb451691ea9d49793a38e1 P 87f7e6a3e9ce4539a39f39e1a116ba76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 885f207e917042ff9d0e5f53807a0ba6 in term 1.
I20260926 09:32:30.190371 1177 leader_election.cc:302] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87f7e6a3e9ce4539a39f39e1a116ba76, 885f207e917042ff9d0e5f53807a0ba6; no voters:
I20260926 09:32:30.190495 1443 raft_consensus.cc:2781] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:30.190671 1443 raft_consensus.cc:686] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 885f207e917042ff9d0e5f53807a0ba6, State: Running, Role: LEADER
I20260926 09:32:30.190785 1443 consensus_queue.cc:237] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } }
I20260926 09:32:30.191685 961 catalog_manager.cc:5495] T 3a0054685ceb451691ea9d49793a38e1 P 885f207e917042ff9d0e5f53807a0ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 885f207e917042ff9d0e5f53807a0ba6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "885f207e917042ff9d0e5f53807a0ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885f207e917042ff9d0e5f53807a0ba6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f99111328ca4d8bb3767978cdfe6528" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f7e6a3e9ce4539a39f39e1a116ba76" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } health_report { overall_health: UNKNOWN } } }
Sep 26 09:32:30 dist-test-slave-21h8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:30.275223 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:30.267203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.0.98.1:40547)
0926 09:32:30.267282 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:30.267285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:30.267421 (+ 136us) server_negotiation.cc:407] Connection header received
0926 09:32:30.267508 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:30.267509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:30.267521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:30.267596 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:30.268180 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.268726 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.269473 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.269667 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.269742 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:30.272166 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:30.272180 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:30.272180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:30.272191 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:30.273919 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:30.274335 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:30.274339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:30.274339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:30.274406 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:30.274624 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:30.274625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:30.274625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:30.274784 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:30.274943 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:30.275010 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 09:32:30.275101 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 09:32:30 dist-test-slave-21h8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415150, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:32:30 dist-test-slave-21h8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415150, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:30.301115 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:30.291221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.0.98.1:40547)
0926 09:32:30.291289 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:30.291291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:30.291434 (+ 143us) server_negotiation.cc:407] Connection header received
0926 09:32:30.291531 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:30.291532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:30.291545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:30.291611 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:30.292413 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.292889 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.293665 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.293843 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.293892 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:30.296725 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:30.296727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:30.296727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:30.296735 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:30.299545 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:30.300082 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:30.300084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:30.300084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:30.300158 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:30.300430 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:30.300431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:30.300431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:30.300562 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:30.300664 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:30.300910 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 09:32:30.301009 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 09:32:30 dist-test-slave-21h8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:32:30 dist-test-slave-21h8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:30.327665 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:30.319094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.0.98.1:40547)
0926 09:32:30.319141 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:30.319144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:30.319214 (+ 70us) server_negotiation.cc:407] Connection header received
0926 09:32:30.319369 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:30.319369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:30.319381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:30.319471 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:30.320250 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.321006 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.322033 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:30.322351 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:30.322436 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:30.325147 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:30.325148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:30.325148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:30.325155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:30.326610 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:30.326950 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:30.326950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:30.326950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:30.327001 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:30.327169 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:30.327170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:30.327170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:30.327272 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:30.327410 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:30.327503 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 09:32:30.327564 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 09:32:30.328173 1067 tablet_service.cc:2908] Rejecting scan request for tablet 3a0054685ceb451691ea9d49793a38e1: Uninitialized: clean time has not yet been initialized
I20260926 09:32:30.328989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 1024
I20260926 09:32:30.334064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 1158
I20260926 09:32:30.338420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 1292
I20260926 09:32:30.343253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 930
2026-09-26T09:32:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2340 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:31 dist-test-slave-21h8 krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:31 dist-test-slave-21h8 krb5kdc[1463](info): set up 2 sockets
Sep 26 09:32:31 dist-test-slave-21h8 krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260926 09:32:32.747076 392 mini_kdc.cc:121] Time spent starting KDC: real 2.381s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:32 dist-test-slave-21h8 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:32.886559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42255
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:32.969408 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:32.969645 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:32.969668 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:32.970835 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:32.970883 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:32:32.970898 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:32.970911 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:32.970924 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:32.970938 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:32.972682 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42255
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:32.972983 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:32.973234 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:32.978319 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:32.978380 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:32.978410 1476 server_base.cc:1034] running on GCE node
W20260926 09:32:32.978382 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:32.978809 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:32.979131 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:32.980301 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790415152980282 us; error 43 us; skew 500 ppm
Sep 26 09:32:32 dist-test-slave-21h8 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:32.989562 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:32.990105 1476 webserver.cc:460] Webserver started at http://127.0.98.62:41213/ using document root <none> and password file <none>
I20260926 09:32:32.990357 1476 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:32.990437 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:32.990587 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:32.991835 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "2d87f0db179347fd8ad22dbcc4244e08"
format_stamp: "Formatted at 2026-09-26 09:32:32 on dist-test-slave-21h8"
server_key: "03cd9c9fce30287108b8844085c56ba1"
server_key_iv: "ca008e9f14a580339a30441d9e9db8ee"
server_key_version: "encryptionkey@0"
I20260926 09:32:32.992234 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d87f0db179347fd8ad22dbcc4244e08"
format_stamp: "Formatted at 2026-09-26 09:32:32 on dist-test-slave-21h8"
server_key: "03cd9c9fce30287108b8844085c56ba1"
server_key_iv: "ca008e9f14a580339a30441d9e9db8ee"
server_key_version: "encryptionkey@0"
I20260926 09:32:32.993629 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:32.994372 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:32.994575 1476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:32.994684 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "2d87f0db179347fd8ad22dbcc4244e08"
format_stamp: "Formatted at 2026-09-26 09:32:32 on dist-test-slave-21h8"
server_key: "03cd9c9fce30287108b8844085c56ba1"
server_key_iv: "ca008e9f14a580339a30441d9e9db8ee"
server_key_version: "encryptionkey@0"
I20260926 09:32:32.994760 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:33.020615 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:33.023007 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:33.023187 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:33.028016 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42255
I20260926 09:32:33.028072 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42255 every 8 connection(s)
I20260926 09:32:33.028455 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:33.029281 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.031821 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08: Bootstrap starting.
I20260926 09:32:33.032009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 1476
I20260926 09:32:33.032138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:33.032361 392 external_mini_cluster.cc:1428] Setting key 29e7b6b5e41a025b2292ae6aafef418b
I20260926 09:32:33.032675 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.033038 1554 log.cc:824] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:33.033797 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08: No bootstrap required, opened a new log
I20260926 09:32:33.035787 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d87f0db179347fd8ad22dbcc4244e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } }
I20260926 09:32:33.035964 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.036005 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d87f0db179347fd8ad22dbcc4244e08, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.036166 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d87f0db179347fd8ad22dbcc4244e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } }
I20260926 09:32:33.036253 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:33.036299 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:33.036350 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:33.036937 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d87f0db179347fd8ad22dbcc4244e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } }
I20260926 09:32:33.037065 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d87f0db179347fd8ad22dbcc4244e08; no voters:
I20260926 09:32:33.037282 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:33.037315 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:33.037436 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [term 1 LEADER]: Becoming Leader. State: Replica: 2d87f0db179347fd8ad22dbcc4244e08, State: Running, Role: LEADER
I20260926 09:32:33.037576 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:33.037585 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d87f0db179347fd8ad22dbcc4244e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } }
I20260926 09:32:33.038015 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d87f0db179347fd8ad22dbcc4244e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d87f0db179347fd8ad22dbcc4244e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } } }
I20260926 09:32:33.038143 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:33.038336 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d87f0db179347fd8ad22dbcc4244e08. Latest consensus state: current_term: 1 leader_uuid: "2d87f0db179347fd8ad22dbcc4244e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d87f0db179347fd8ad22dbcc4244e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42255 } } }
I20260926 09:32:33.038425 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:33.038463 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:33.039003 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:33.040978 1568 catalog_manager.cc:1347] Generated new cluster ID: ca984b0ebbe34c1c88bc8999cb6571ab
I20260926 09:32:33.041036 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:33.044791 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.032944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.0.98.62:42255)
0926 09:32:33.033124 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.033134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.033171 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:32:33.033795 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.033803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.033859 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.034041 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.034721 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.035710 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.036755 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.037300 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.037489 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.041275 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.041280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.041280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.041293 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.043094 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.043573 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.043577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.043577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.043684 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.043914 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.043915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.043916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.044169 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.044349 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.044421 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.044567 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260926 09:32:33.050303 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:33.051070 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:33.053711 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08: Generated new TSK 0
I20260926 09:32:33.053927 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:32:33.088256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:33.173882 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:33.174106 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:33.174131 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:33.175398 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:33.175442 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:33.175464 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:32:33.175513 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:33.177083 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:33.177369 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:33.177623 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:33.182884 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:33.182883 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:33.182885 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:33.183334 1580 server_base.cc:1034] running on GCE node
I20260926 09:32:33.183542 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:33.183811 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:33.185019 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790415153185006 us; error 28 us; skew 500 ppm
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:33.187660 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:33.188126 1580 webserver.cc:460] Webserver started at http://127.0.98.1:42065/ using document root <none> and password file <none>
I20260926 09:32:33.188328 1580 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:33.188370 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:33.188460 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:33.189504 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "0cf1fd75a5c742aaab1053dc5bf61acd"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "58738111731553507b02e241e39958f6"
server_key_iv: "447a366105acb4df45d05c0721446f18"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.189815 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0cf1fd75a5c742aaab1053dc5bf61acd"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "58738111731553507b02e241e39958f6"
server_key_iv: "447a366105acb4df45d05c0721446f18"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.191105 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:33.191900 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.192097 1580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:32:33.192184 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "0cf1fd75a5c742aaab1053dc5bf61acd"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "58738111731553507b02e241e39958f6"
server_key_iv: "447a366105acb4df45d05c0721446f18"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.192260 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:33.206547 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:33.208878 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:33.209151 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:33.209728 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:33.209771 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.209800 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:33.209854 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.216991 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36877
I20260926 09:32:33.217070 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36877 every 8 connection(s)
I20260926 09:32:33.217442 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:33.220701 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.210305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35087 (local address 127.0.98.62:42255)
0926 09:32:33.210370 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.210372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.210592 (+ 220us) server_negotiation.cc:407] Connection header received
0926 09:32:33.211163 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.211164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.211178 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.211248 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.212348 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.213279 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.215036 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.215254 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.215330 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.218185 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.218187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.218187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.218193 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.219486 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.219862 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.219863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.219864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.219926 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.220138 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.220139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.220139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.220244 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.220293 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.220551 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.220626 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:32:33.220922 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.210313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.1:35087)
0926 09:32:33.210465 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.210503 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.211046 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.211416 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.211422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.211853 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:33.212227 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.212235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.213386 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.213388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.214935 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.214938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.215381 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.215382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.215546 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.216235 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:33.216248 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:33.218094 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:33.219558 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.219564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.219566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.219795 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.219986 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.219986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.219986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.220085 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.220302 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:33.220306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:33.220440 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.220526 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.220665 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260926 09:32:33.221539 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42255
I20260926 09:32:33.221628 1711 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:33.221832 1711 heartbeater.cc:511] Master 127.0.98.62:42255 requested a full tablet report, sending...
I20260926 09:32:33.222419 1508 ts_manager.cc:194] Registered new tserver with Master: 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877)
I20260926 09:32:33.223141 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35087
I20260926 09:32:33.223536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 1580
I20260926 09:32:33.223625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:33.223814 392 external_mini_cluster.cc:1428] Setting key 7259ab3b593f797a5128c86bc9b372dc
I20260926 09:32:33.229157 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.222745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44357 (local address 127.0.98.62:42255)
0926 09:32:33.222789 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.222791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.223468 (+ 677us) server_negotiation.cc:407] Connection header received
0926 09:32:33.223529 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.223530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.223542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.223599 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.223932 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.224580 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.225299 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.225517 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.225595 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.226594 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.226596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.226596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.226602 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.228017 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.228443 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.228444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.228444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.228494 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.228708 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.228709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.228709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.228807 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.228856 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.229036 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.229092 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:33.229310 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.222680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.1:44357)
0926 09:32:33.223386 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.223399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.223466 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.223603 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.223604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.223790 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:33.223840 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.223842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.224681 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.224682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.225213 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.225215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.225593 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.225594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.225661 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.226162 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:33.226167 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:33.226501 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:33.228104 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.228105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.228106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.228361 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.228562 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.228563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.228564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.228643 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.228873 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:33.228874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:33.228928 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.229018 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.229147 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:33.251260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:33.338862 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:33.339104 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:33.339144 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:33.340492 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:33.340538 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:33.340560 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:32:33.340628 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:33.342160 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:33.342422 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:33.342718 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:33.348724 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:33.348726 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:33.349079 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:33.349447 1720 server_base.cc:1034] running on GCE node
I20260926 09:32:33.349670 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:33.349947 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:33.351121 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790415153351102 us; error 32 us; skew 500 ppm
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:33.353681 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:33.354132 1720 webserver.cc:460] Webserver started at http://127.0.98.2:33807/ using document root <none> and password file <none>
I20260926 09:32:33.354336 1720 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:33.354382 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:33.354470 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:33.355587 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "72cdbea125ff4a57a8c991426bce0621"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "8bf36c359cf202cb841756ec705c8008"
server_key_iv: "1fa5e192dcc9a27fb3f759a919f4a10f"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.355962 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72cdbea125ff4a57a8c991426bce0621"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "8bf36c359cf202cb841756ec705c8008"
server_key_iv: "1fa5e192dcc9a27fb3f759a919f4a10f"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.357473 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:33.358455 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.358691 1720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:33.358793 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "72cdbea125ff4a57a8c991426bce0621"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "8bf36c359cf202cb841756ec705c8008"
server_key_iv: "1fa5e192dcc9a27fb3f759a919f4a10f"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.358877 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:33.375171 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:33.377159 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:33.377321 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:33.377925 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:33.377990 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.378036 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:33.378065 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:33.385912 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44781
I20260926 09:32:33.386003 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44781 every 8 connection(s)
I20260926 09:32:33.386390 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:33.389020 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.378338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57165 (local address 127.0.98.62:42255)
0926 09:32:33.378426 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.378429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.378715 (+ 286us) server_negotiation.cc:407] Connection header received
0926 09:32:33.379531 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.379533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.379547 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.379606 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.380579 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.381130 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.382229 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.382446 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.382525 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.385669 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.385671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.385672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.385682 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.387503 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.387949 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.387950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.387951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.388026 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.388291 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.388297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.388297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.388453 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.388532 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.388821 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.388918 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:32:33.389245 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.378343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.2:57165)
0926 09:32:33.378543 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.378600 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.379405 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.379710 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.379716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.380159 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:33.380488 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.380494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.381244 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.381245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.382125 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.382128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.382545 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.382546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.382720 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.383640 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:33.383649 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:33.385529 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:33.387606 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.387613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.387615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.387862 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.388102 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.388103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.388104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.388216 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.388522 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:33.388528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:33.388686 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.388800 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.388975 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260926 09:32:33.389720 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42255
I20260926 09:32:33.389811 1851 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:33.390023 1851 heartbeater.cc:511] Master 127.0.98.62:42255 requested a full tablet report, sending...
I20260926 09:32:33.390483 1507 ts_manager.cc:194] Registered new tserver with Master: 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781)
I20260926 09:32:33.390952 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57165
I20260926 09:32:33.396498 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.390808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.2:46275)
0926 09:32:33.390938 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.390949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.391035 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.391184 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.391185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.391431 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:33.391494 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.391496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.392198 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.392199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.392734 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.392736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.393111 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.393112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.393180 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.393591 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:33.393595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:33.393889 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:33.395356 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.395358 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.395358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.395534 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.395746 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.395747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.395747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.395813 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.396135 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:33.396136 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:33.396180 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.396269 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.396383 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260926 09:32:33.396489 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.390882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46275 (local address 127.0.98.62:42255)
0926 09:32:33.390928 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.390929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.391005 (+ 76us) server_negotiation.cc:407] Connection header received
0926 09:32:33.391106 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.391107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.391117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.391180 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.391600 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.392096 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.392822 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.393037 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.393113 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.393996 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.393998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.393998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.394006 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.395246 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.395608 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.395609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.395609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.395669 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.395881 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.395882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.395883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.396036 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.396116 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.396297 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.396393 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:32:33.396514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 1720
I20260926 09:32:33.396621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:33.396819 392 external_mini_cluster.cc:1428] Setting key a1d9461fb6d828e1ae3d7cc65a76aa22
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:33.422645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42255
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:33.505934 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:33.506192 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:33.506245 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:33.507546 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:33.507609 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:33.507643 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:32:33.507699 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:33.509268 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:33.509538 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:33.509773 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:33.514985 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:33.515074 1860 server_base.cc:1034] running on GCE node
W20260926 09:32:33.514992 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:33.514995 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:33.515431 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:33.515641 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:33.516795 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790415153516786 us; error 32 us; skew 500 ppm
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:33.519585 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:33.520074 1860 webserver.cc:460] Webserver started at http://127.0.98.3:42691/ using document root <none> and password file <none>
I20260926 09:32:33.520323 1860 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:33.520368 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:33.520539 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:33.521631 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b6d9b67235e4dcca250ed38dbed9d4b"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "f7ad3881936e1f85453a4eda3697f90b"
server_key_iv: "cb24cf37e15a5721a81c89d05bbd7d02"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.522059 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b6d9b67235e4dcca250ed38dbed9d4b"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "f7ad3881936e1f85453a4eda3697f90b"
server_key_iv: "cb24cf37e15a5721a81c89d05bbd7d02"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.523463 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:33.524233 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.524443 1860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:33.524525 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "0b6d9b67235e4dcca250ed38dbed9d4b"
format_stamp: "Formatted at 2026-09-26 09:32:33 on dist-test-slave-21h8"
server_key: "f7ad3881936e1f85453a4eda3697f90b"
server_key_iv: "cb24cf37e15a5721a81c89d05bbd7d02"
server_key_version: "encryptionkey@0"
I20260926 09:32:33.524580 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:33.543856 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:33.546191 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:33.546308 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:33.546795 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:33.546833 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.546866 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:33.546927 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.553911 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41211
I20260926 09:32:33.554005 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41211 every 8 connection(s)
I20260926 09:32:33.554335 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:33.557576 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.547352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52785 (local address 127.0.98.62:42255)
0926 09:32:33.547430 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.547432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.547661 (+ 229us) server_negotiation.cc:407] Connection header received
0926 09:32:33.548181 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.548183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.548198 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.548283 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.549203 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.550060 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.551807 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.552137 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.552235 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.555001 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.555002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.555002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.555009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.556268 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.556648 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.556649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.556650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.556700 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.556938 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.556939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.556939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.557049 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.557106 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.557437 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.557508 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:32:33.557689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 1860
I20260926 09:32:33.557772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:33.557946 392 external_mini_cluster.cc:1428] Setting key dd8712abb94435af6f1064f01cbdd321
I20260926 09:32:33.557943 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.547357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.3:52785)
0926 09:32:33.547525 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.547567 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.548082 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.548362 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.548366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.548731 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:33.549098 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.549106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.550176 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.550177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.551665 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.551668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.552237 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.552238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.552348 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.553048 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:33.553058 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:33.554898 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:33.556343 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.556349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.556351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.556577 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.556761 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.556762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.556762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.556863 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.557146 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:33.557152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:33.557300 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.557412 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.557649 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260926 09:32:33.558493 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42255
I20260926 09:32:33.558604 1991 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:33.558826 1991 heartbeater.cc:511] Master 127.0.98.62:42255 requested a full tablet report, sending...
I20260926 09:32:33.559234 1508 ts_manager.cc:194] Registered new tserver with Master: 0b6d9b67235e4dcca250ed38dbed9d4b (127.0.98.3:41211)
I20260926 09:32:33.559799 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52785
I20260926 09:32:33.565419 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.559781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48619 (local address 127.0.98.62:42255)
0926 09:32:33.559823 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.559825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.559910 (+ 85us) server_negotiation.cc:407] Connection header received
0926 09:32:33.560005 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.560006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.560017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.560080 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.560431 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.560939 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.561723 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.561918 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.562012 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.563003 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.563004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.563005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.563012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.564232 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.564592 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.564594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.564594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.564654 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.564855 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.564855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.564856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.564971 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.565047 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.565240 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.565313 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:33.565430 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.559689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.98.3:48619)
0926 09:32:33.559834 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.559846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.559942 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.560085 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.560086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.560286 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:33.560344 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.560346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.561048 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.561049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.561578 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.561580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.562014 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.562015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.562091 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.562544 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:33.562549 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:33.562915 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:33.564320 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.564321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.564321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.564500 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.564722 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.564722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.564723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.564789 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.565052 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:33.565053 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:33.565099 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.565179 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.565313 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260926 09:32:33.569981 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:33.592466 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.585299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.0.98.62:42255)
0926 09:32:33.585391 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.585393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.585581 (+ 188us) server_negotiation.cc:407] Connection header received
0926 09:32:33.585614 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.585615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.585627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.585696 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.586369 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.586914 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.587702 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.587881 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.587944 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.590206 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.590207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.590207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.590214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.591556 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.591853 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.591853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.591854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.591919 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.592079 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.592080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.592080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.592183 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.592254 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.592342 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.592396 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 09:32:33.594153 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:33.595944 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:33.602695 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.598347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44781 (local address 127.0.0.1:36848)
0926 09:32:33.598643 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.598690 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.598757 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.599026 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.599029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.599031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:33.599100 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.599102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.600217 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.600217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.601475 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.601477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.602343 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.602344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.602437 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.602452 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.602541 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.602600 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":51,"threads_started":1}
I20260926 09:32:33.602705 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.598462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.0.98.2:44781)
0926 09:32:33.598854 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.598859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.598877 (+ 18us) server_negotiation.cc:407] Connection header received
0926 09:32:33.598910 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.598912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.598925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.599007 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:33.599171 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.600129 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.601567 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.602256 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.602344 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.602428 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.602569 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.602605 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":81,"threads_started":1}
I20260926 09:32:33.603155 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.598355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36877 (local address 127.0.0.1:57548)
0926 09:32:33.598660 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.598691 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.598756 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.599624 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.599625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.599627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:33.599667 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.599668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.600920 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.600921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.601933 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.601936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.602885 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.602886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.602953 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.602957 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.603018 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.603059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":73,"threads_started":1}
I20260926 09:32:33.603189 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.598564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.0.98.1:36877)
0926 09:32:33.599467 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.599471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.599481 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:33.599518 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.599520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.599534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.599614 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:33.599724 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.600828 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.602024 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.602712 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.602772 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.602845 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.603052 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.603096 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":70,"threads_started":1}
I20260926 09:32:33.603538 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.598635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.0.98.3:41211)
0926 09:32:33.598810 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.598814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.599215 (+ 401us) server_negotiation.cc:407] Connection header received
0926 09:32:33.599259 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.599262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.599277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.599382 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:33.599517 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.600630 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.601877 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.602612 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.602685 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.603185 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.603390 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.603436 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260926 09:32:33.603156 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.598535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41211 (local address 127.0.0.1:52096)
0926 09:32:33.599100 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.599111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.599159 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.599385 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.599386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.599389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:33.599443 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.599445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.600728 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.600729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.601790 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.601792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.602922 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.602923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.602987 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.602990 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.603032 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.603059 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":86,"thread_start_us":71,"threads_started":1}
I20260926 09:32:33.603852 1785 tablet_service.cc:1467] Processing CreateTablet for tablet e7a5305c72924361a524991459031d05 (DEFAULT_TABLE table=test-table [id=4adca4d13a014e7e80e14ef04a8bab91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:33.603852 1627 tablet_service.cc:1467] Processing CreateTablet for tablet e7a5305c72924361a524991459031d05 (DEFAULT_TABLE table=test-table [id=4adca4d13a014e7e80e14ef04a8bab91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:33.604229 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a5305c72924361a524991459031d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.604226 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a5305c72924361a524991459031d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.608180 1925 tablet_service.cc:1467] Processing CreateTablet for tablet e7a5305c72924361a524991459031d05 (DEFAULT_TABLE table=test-table [id=4adca4d13a014e7e80e14ef04a8bab91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:33.608421 2014 tablet_bootstrap.cc:492] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Bootstrap starting.
I20260926 09:32:33.608569 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a5305c72924361a524991459031d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.609131 2014 tablet_bootstrap.cc:654] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.609491 2015 tablet_bootstrap.cc:492] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Bootstrap starting.
I20260926 09:32:33.609503 2014 log.cc:824] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:33.610126 2015 tablet_bootstrap.cc:654] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.610481 2015 log.cc:824] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:33.610605 2014 tablet_bootstrap.cc:492] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: No bootstrap required, opened a new log
I20260926 09:32:33.610767 2014 ts_tablet_manager.cc:1397] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:33.611291 2015 tablet_bootstrap.cc:492] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: No bootstrap required, opened a new log
I20260926 09:32:33.611389 2015 ts_tablet_manager.cc:1397] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 09:32:33.612406 2014 raft_consensus.cc:348] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.612617 2014 raft_consensus.cc:374] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.612664 2014 raft_consensus.cc:729] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf1fd75a5c742aaab1053dc5bf61acd, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.612767 2018 tablet_bootstrap.cc:492] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Bootstrap starting.
I20260926 09:32:33.612784 2014 consensus_queue.cc:260] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.613080 2014 ts_tablet_manager.cc:1428] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:33.613210 1711 heartbeater.cc:503] Master 127.0.98.62:42255 was elected leader, sending a full tablet report...
I20260926 09:32:33.613441 2018 tablet_bootstrap.cc:654] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.613626 2015 raft_consensus.cc:348] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.613742 2018 log.cc:824] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:33.613786 2015 raft_consensus.cc:374] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.613819 2015 raft_consensus.cc:729] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72cdbea125ff4a57a8c991426bce0621, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.613950 2015 consensus_queue.cc:260] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.614216 2015 ts_tablet_manager.cc:1428] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:32:33.614308 1851 heartbeater.cc:503] Master 127.0.98.62:42255 was elected leader, sending a full tablet report...
I20260926 09:32:33.614506 2018 tablet_bootstrap.cc:492] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: No bootstrap required, opened a new log
I20260926 09:32:33.614568 2018 ts_tablet_manager.cc:1397] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:33.616170 2018 raft_consensus.cc:348] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.616338 2018 raft_consensus.cc:374] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.616362 2018 raft_consensus.cc:729] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6d9b67235e4dcca250ed38dbed9d4b, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.616467 2018 consensus_queue.cc:260] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.616674 2018 ts_tablet_manager.cc:1428] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:33.616726 1991 heartbeater.cc:503] Master 127.0.98.62:42255 was elected leader, sending a full tablet report...
W20260926 09:32:33.637300 1852 tablet.cc:2369] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:32:33.718412 1712 tablet.cc:2369] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:32:33.804965 1992 tablet.cc:2369] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:32:33.825665 2022 raft_consensus.cc:482] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:33.825800 2022 raft_consensus.cc:504] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.826153 2022 leader_election.cc:288] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781), 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877)
I20260926 09:32:33.829475 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.826518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57679 (local address 127.0.98.1:36877)
0926 09:32:33.826567 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.826569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.826709 (+ 140us) server_negotiation.cc:407] Connection header received
0926 09:32:33.826779 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.826781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.826793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.826888 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:33.827058 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.827602 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.828609 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.829101 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.829165 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.829177 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.829354 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.829398 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:32:33.829483 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.826395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36877 (local address 127.0.98.3:57679)
0926 09:32:33.826631 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.826640 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.826722 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.826896 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.826897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.826910 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:33.826984 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.826986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.827714 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.827715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.828523 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.828525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.829166 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.829167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.829245 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.829249 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.829335 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.829385 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":61,"thread_start_us":116,"threads_started":1}
I20260926 09:32:33.829646 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.826419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58193 (local address 127.0.98.2:44781)
0926 09:32:33.826470 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.826473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.826509 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:32:33.826623 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.826624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.826634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.826705 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:33.826860 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.827480 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.828572 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.829098 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.829183 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.829194 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.829526 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.829574 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:32:33.829499 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.826389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44781 (local address 127.0.98.3:58193)
0926 09:32:33.826460 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.826472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.826529 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.826711 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.826712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.826713 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:33.826789 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.826790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.827565 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.827566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.828484 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.828486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.829227 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.829228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.829302 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.829305 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.829351 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.829394 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 09:32:33.830080 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a5305c72924361a524991459031d05" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" is_pre_election: true
I20260926 09:32:33.830216 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a5305c72924361a524991459031d05" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cdbea125ff4a57a8c991426bce0621" is_pre_election: true
I20260926 09:32:33.830257 1665 raft_consensus.cc:2454] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 0.
I20260926 09:32:33.830346 1805 raft_consensus.cc:2454] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 0.
I20260926 09:32:33.830787 1878 leader_election.cc:302] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6d9b67235e4dcca250ed38dbed9d4b, 0cf1fd75a5c742aaab1053dc5bf61acd; no voters:
I20260926 09:32:33.830935 2022 raft_consensus.cc:2781] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:33.830991 2022 raft_consensus.cc:482] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:33.831014 2022 raft_consensus.cc:3037] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:33.831782 2022 raft_consensus.cc:504] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.831957 2022 leader_election.cc:288] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 election: Requested vote from peers 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781), 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877)
I20260926 09:32:33.832098 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a5305c72924361a524991459031d05" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cdbea125ff4a57a8c991426bce0621"
I20260926 09:32:33.832124 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a5305c72924361a524991459031d05" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd"
I20260926 09:32:33.832190 1665 raft_consensus.cc:3037] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:33.832181 1805 raft_consensus.cc:3037] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:33.833021 1665 raft_consensus.cc:2454] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 1.
I20260926 09:32:33.833066 1805 raft_consensus.cc:2454] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 1.
I20260926 09:32:33.833191 1878 leader_election.cc:302] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6d9b67235e4dcca250ed38dbed9d4b, 0cf1fd75a5c742aaab1053dc5bf61acd; no voters:
I20260926 09:32:33.833280 2022 raft_consensus.cc:2781] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:33.833418 2022 raft_consensus.cc:686] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 LEADER]: Becoming Leader. State: Replica: 0b6d9b67235e4dcca250ed38dbed9d4b, State: Running, Role: LEADER
I20260926 09:32:33.833521 2022 consensus_queue.cc:237] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.834388 1508 catalog_manager.cc:5495] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6d9b67235e4dcca250ed38dbed9d4b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260926 09:32:33.890823 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.887941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.0.98.3:41211)
0926 09:32:33.888005 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.888008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.888056 (+ 48us) server_negotiation.cc:407] Connection header received
0926 09:32:33.888172 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.888173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.888199 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.888277 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:33.888449 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.888927 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.889923 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.890136 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.890213 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.890461 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:33.890565 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:33.890654 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.890706 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.890729 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:32:33.892777 1805 raft_consensus.cc:1261] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Refusing update from remote peer 0b6d9b67235e4dcca250ed38dbed9d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:33.892860 1665 raft_consensus.cc:1261] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Refusing update from remote peer 0b6d9b67235e4dcca250ed38dbed9d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:33.893123 2022 consensus_queue.cc:1035] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:33.893306 2024 consensus_queue.cc:1035] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:33.895980 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333540470341869568 to 7333540470102716416. Current Snapshot: MvccSnapshot[applied={T|T < 7333540470341869568 or (T in {7333540470341869568})}]
I20260926 09:32:33.898942 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:33.899075 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:33.899946 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08: Sending DeleteTablet for 3 replicas of tablet e7a5305c72924361a524991459031d05
I20260926 09:32:33.900269 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a5305c72924361a524991459031d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36848
I20260926 09:32:33.900295 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a5305c72924361a524991459031d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52096
I20260926 09:32:33.900525 1627 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a5305c72924361a524991459031d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57548
I20260926 09:32:33.900583 2037 tablet_replica.cc:330] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: stopping tablet replica
I20260926 09:32:33.900666 2038 tablet_replica.cc:330] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: stopping tablet replica
I20260926 09:32:33.900692 2037 raft_consensus.cc:2229] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:33.900707 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 09:32:33.900792 2037 raft_consensus.cc:2258] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 09:32:33.900853 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:33.901008 2038 raft_consensus.cc:2229] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:33.901166 2038 raft_consensus.cc:2258] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:33.901254 2037 ts_tablet_manager.cc:1905] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:33.901722 2038 ts_tablet_manager.cc:1905] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:33.902151 1925 tablet_service.cc:1467] Processing CreateTablet for tablet fc1d0782e61841f0992c655380285a3f (DEFAULT_TABLE table=test-table [id=f8770054727e41618de2d6d853932433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:33.902271 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1d0782e61841f0992c655380285a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.903193 2037 ts_tablet_manager.cc:1918] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:33.903272 2037 log.cc:1196] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/e7a5305c72924361a524991459031d05
I20260926 09:32:33.903460 2018 tablet_bootstrap.cc:492] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: Bootstrap starting.
I20260926 09:32:33.903565 2037 ts_tablet_manager.cc:1939] T e7a5305c72924361a524991459031d05 P 72cdbea125ff4a57a8c991426bce0621: Deleting consensus metadata
I20260926 09:32:33.903753 1627 tablet_service.cc:1467] Processing CreateTablet for tablet fc1d0782e61841f0992c655380285a3f (DEFAULT_TABLE table=test-table [id=f8770054727e41618de2d6d853932433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:33.903863 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1d0782e61841f0992c655380285a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.903896 2018 tablet_bootstrap.cc:654] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.903971 1495 catalog_manager.cc:4841] TS 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781): tablet e7a5305c72924361a524991459031d05 (table test-table [id=4adca4d13a014e7e80e14ef04a8bab91]) successfully deleted
I20260926 09:32:33.904503 2018 tablet_bootstrap.cc:492] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: No bootstrap required, opened a new log
I20260926 09:32:33.904556 2018 ts_tablet_manager.cc:1397] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:33.904633 2038 ts_tablet_manager.cc:1918] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:33.904691 2018 raft_consensus.cc:348] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.904778 2018 raft_consensus.cc:374] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.904805 2018 raft_consensus.cc:729] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6d9b67235e4dcca250ed38dbed9d4b, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.904753 2038 log.cc:1196] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/e7a5305c72924361a524991459031d05
I20260926 09:32:33.904860 2018 consensus_queue.cc:260] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.904970 2018 ts_tablet_manager.cc:1428] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:33.905375 1785 tablet_service.cc:1467] Processing CreateTablet for tablet fc1d0782e61841f0992c655380285a3f (DEFAULT_TABLE table=test-table [id=f8770054727e41618de2d6d853932433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:33.905403 2014 tablet_bootstrap.cc:492] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: Bootstrap starting.
I20260926 09:32:33.905484 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1d0782e61841f0992c655380285a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.905622 2038 ts_tablet_manager.cc:1939] T e7a5305c72924361a524991459031d05 P 0b6d9b67235e4dcca250ed38dbed9d4b: Deleting consensus metadata
I20260926 09:32:33.905798 2014 tablet_bootstrap.cc:654] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.906453 2014 tablet_bootstrap.cc:492] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: No bootstrap required, opened a new log
I20260926 09:32:33.906509 2014 ts_tablet_manager.cc:1397] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:33.906674 2014 raft_consensus.cc:348] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.906746 2014 raft_consensus.cc:374] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.906740 2015 tablet_bootstrap.cc:492] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: Bootstrap starting.
I20260926 09:32:33.906771 2014 raft_consensus.cc:729] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf1fd75a5c742aaab1053dc5bf61acd, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.906816 2014 consensus_queue.cc:260] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.906929 2014 ts_tablet_manager.cc:1428] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:33.907133 2015 tablet_bootstrap.cc:654] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.907137 2039 tablet_replica.cc:330] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: stopping tablet replica
I20260926 09:32:33.907204 1495 catalog_manager.cc:4841] TS 0b6d9b67235e4dcca250ed38dbed9d4b (127.0.98.3:41211): tablet e7a5305c72924361a524991459031d05 (table test-table [id=4adca4d13a014e7e80e14ef04a8bab91]) successfully deleted
I20260926 09:32:33.907258 2039 raft_consensus.cc:2229] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:33.907407 2039 raft_consensus.cc:2258] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:33.907835 2015 tablet_bootstrap.cc:492] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: No bootstrap required, opened a new log
I20260926 09:32:33.907900 2015 ts_tablet_manager.cc:1397] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:33.908057 2039 ts_tablet_manager.cc:1905] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:33.908049 2015 raft_consensus.cc:348] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.908114 2015 raft_consensus.cc:374] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.908138 2015 raft_consensus.cc:729] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72cdbea125ff4a57a8c991426bce0621, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.908180 2015 consensus_queue.cc:260] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.908510 2015 ts_tablet_manager.cc:1428] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:33.909451 2039 ts_tablet_manager.cc:1918] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:33.909536 2039 log.cc:1196] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/e7a5305c72924361a524991459031d05
I20260926 09:32:33.910019 2039 ts_tablet_manager.cc:1939] T e7a5305c72924361a524991459031d05 P 0cf1fd75a5c742aaab1053dc5bf61acd: Deleting consensus metadata
I20260926 09:32:33.910418 1494 catalog_manager.cc:4841] TS 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877): tablet e7a5305c72924361a524991459031d05 (table test-table [id=4adca4d13a014e7e80e14ef04a8bab91]) successfully deleted
I20260926 09:32:33.922506 2032 raft_consensus.cc:482] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:33.922621 2032 raft_consensus.cc:504] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.922818 2032 leader_election.cc:288] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877), 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781)
I20260926 09:32:33.923005 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1d0782e61841f0992c655380285a3f" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cdbea125ff4a57a8c991426bce0621" is_pre_election: true
I20260926 09:32:33.923002 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1d0782e61841f0992c655380285a3f" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" is_pre_election: true
I20260926 09:32:33.923122 1665 raft_consensus.cc:2454] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 0.
I20260926 09:32:33.923125 1805 raft_consensus.cc:2454] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 0.
I20260926 09:32:33.923290 1879 leader_election.cc:302] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6d9b67235e4dcca250ed38dbed9d4b, 72cdbea125ff4a57a8c991426bce0621; no voters:
I20260926 09:32:33.923475 2032 raft_consensus.cc:2781] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:33.923512 2032 raft_consensus.cc:482] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:33.923537 2032 raft_consensus.cc:3037] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:33.924383 2032 raft_consensus.cc:504] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.924537 2032 leader_election.cc:288] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 election: Requested vote from peers 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877), 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781)
I20260926 09:32:33.924683 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1d0782e61841f0992c655380285a3f" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd"
I20260926 09:32:33.924768 1665 raft_consensus.cc:3037] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:33.924746 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1d0782e61841f0992c655380285a3f" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cdbea125ff4a57a8c991426bce0621"
I20260926 09:32:33.924815 1805 raft_consensus.cc:3037] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:33.925554 1805 raft_consensus.cc:2454] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 1.
I20260926 09:32:33.925555 1665 raft_consensus.cc:2454] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 1.
I20260926 09:32:33.925724 1878 leader_election.cc:302] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6d9b67235e4dcca250ed38dbed9d4b, 0cf1fd75a5c742aaab1053dc5bf61acd; no voters:
I20260926 09:32:33.925827 2032 raft_consensus.cc:2781] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:33.925889 2032 raft_consensus.cc:686] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 LEADER]: Becoming Leader. State: Replica: 0b6d9b67235e4dcca250ed38dbed9d4b, State: Running, Role: LEADER
I20260926 09:32:33.925941 2032 consensus_queue.cc:237] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.926559 1507 catalog_manager.cc:5495] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6d9b67235e4dcca250ed38dbed9d4b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } health_report { overall_health: HEALTHY } } }
Sep 26 09:32:33 dist-test-slave-21h8 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:33.971937 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.964056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42255 (local address 127.0.0.1:51946)
0926 09:32:33.964244 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:33.964254 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:33.964316 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:33.964477 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:33.964478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:33.964780 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:33.964835 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.964837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.965790 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.965791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.966324 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:33.966326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.966773 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.966774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:33.966851 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.967504 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:33.967508 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:33.969080 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:33.970812 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.970819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.970821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.971020 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.971237 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:33.971237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:33.971238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:33.971294 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:33.971643 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:33.971649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:33.971694 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 09:32:33.971749 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:32:33.971845 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260926 09:32:33.972033 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:33.964152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.0.98.62:42255)
0926 09:32:33.964227 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:33.964230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:33.964318 (+ 88us) server_negotiation.cc:407] Connection header received
0926 09:32:33.964367 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:33.964368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:33.964380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:33.964464 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:33.964917 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.965687 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.966412 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:33.966687 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:33.966766 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:33.969175 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:33.969177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:33.969177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:33.969185 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:33.970715 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.971097 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.971098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.971098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.971163 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:33.971431 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:33.971432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:33.971432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:33.971574 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:33.971648 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:33.971800 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 09:32:33.971924 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":42}
I20260926 09:32:33.974198 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51946:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:32:33.974436 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:33.976046 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2cad074a554a85bcba2cf0f71c97d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02369d88539e4f619b361e721f1a30f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:32:33.976087 1627 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2cad074a554a85bcba2cf0f71c97d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02369d88539e4f619b361e721f1a30f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:32:33.976219 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2cad074a554a85bcba2cf0f71c97d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.976218 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2cad074a554a85bcba2cf0f71c97d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.977732 2018 tablet_bootstrap.cc:492] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: Bootstrap starting.
I20260926 09:32:33.977749 2014 tablet_bootstrap.cc:492] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: Bootstrap starting.
I20260926 09:32:33.977818 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2cad074a554a85bcba2cf0f71c97d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02369d88539e4f619b361e721f1a30f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:32:33.977972 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2cad074a554a85bcba2cf0f71c97d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:33.978217 2014 tablet_bootstrap.cc:654] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.978415 2018 tablet_bootstrap.cc:654] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.979108 2018 tablet_bootstrap.cc:492] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: No bootstrap required, opened a new log
I20260926 09:32:33.979161 2018 ts_tablet_manager.cc:1397] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:33.979352 2018 raft_consensus.cc:348] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.979463 2018 raft_consensus.cc:374] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.979485 2018 raft_consensus.cc:729] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6d9b67235e4dcca250ed38dbed9d4b, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.979528 2018 consensus_queue.cc:260] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.979622 2018 ts_tablet_manager.cc:1428] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:33.979725 2032 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:33.979807 2032 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:33.979826 2015 tablet_bootstrap.cc:492] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: Bootstrap starting.
I20260926 09:32:33.980182 2014 tablet_bootstrap.cc:492] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: No bootstrap required, opened a new log
I20260926 09:32:33.980235 2014 ts_tablet_manager.cc:1397] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:32:33.980254 2015 tablet_bootstrap.cc:654] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:33.980378 2014 raft_consensus.cc:348] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.980449 2014 raft_consensus.cc:374] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.980471 2014 raft_consensus.cc:729] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf1fd75a5c742aaab1053dc5bf61acd, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.980517 2014 consensus_queue.cc:260] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.980604 2014 ts_tablet_manager.cc:1428] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.980786 2019 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:33.980883 2019 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:33.981003 2015 tablet_bootstrap.cc:492] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: No bootstrap required, opened a new log
I20260926 09:32:33.981073 2015 ts_tablet_manager.cc:1397] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:33.981220 2015 raft_consensus.cc:348] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.981297 2015 raft_consensus.cc:374] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:33.981323 2015 raft_consensus.cc:729] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72cdbea125ff4a57a8c991426bce0621, State: Initialized, Role: FOLLOWER
I20260926 09:32:33.981375 2015 consensus_queue.cc:260] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:33.981490 2015 ts_tablet_manager.cc:1428] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:33.981572 2021 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:33.981675 2021 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:34.189525 2032 raft_consensus.cc:482] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:34.189640 2032 raft_consensus.cc:504] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:34.189883 2032 leader_election.cc:288] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877), 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781)
I20260926 09:32:34.190237 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2cad074a554a85bcba2cf0f71c97d4" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" is_pre_election: true
I20260926 09:32:34.190776 1665 raft_consensus.cc:2454] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 0.
I20260926 09:32:34.190734 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2cad074a554a85bcba2cf0f71c97d4" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cdbea125ff4a57a8c991426bce0621" is_pre_election: true
I20260926 09:32:34.190825 1805 raft_consensus.cc:2454] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 0.
I20260926 09:32:34.190968 1878 leader_election.cc:302] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6d9b67235e4dcca250ed38dbed9d4b, 0cf1fd75a5c742aaab1053dc5bf61acd; no voters:
I20260926 09:32:34.191112 2032 raft_consensus.cc:2781] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:34.191161 2032 raft_consensus.cc:482] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:34.191193 2032 raft_consensus.cc:3037] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:34.191934 2032 raft_consensus.cc:504] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:34.192096 2032 leader_election.cc:288] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 election: Requested vote from peers 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877), 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781)
I20260926 09:32:34.192270 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2cad074a554a85bcba2cf0f71c97d4" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd"
I20260926 09:32:34.192281 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2cad074a554a85bcba2cf0f71c97d4" candidate_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cdbea125ff4a57a8c991426bce0621"
I20260926 09:32:34.192358 1665 raft_consensus.cc:3037] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:34.192358 1805 raft_consensus.cc:3037] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:34.193524 1805 raft_consensus.cc:2454] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 1.
I20260926 09:32:34.193524 1665 raft_consensus.cc:2454] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6d9b67235e4dcca250ed38dbed9d4b in term 1.
I20260926 09:32:34.193720 1878 leader_election.cc:302] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6d9b67235e4dcca250ed38dbed9d4b, 0cf1fd75a5c742aaab1053dc5bf61acd; no voters:
I20260926 09:32:34.193837 2032 raft_consensus.cc:2781] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:34.193893 2032 raft_consensus.cc:686] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 LEADER]: Becoming Leader. State: Replica: 0b6d9b67235e4dcca250ed38dbed9d4b, State: Running, Role: LEADER
I20260926 09:32:34.193970 2032 consensus_queue.cc:237] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } }
I20260926 09:32:34.194296 2024 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: TxnStatusTablet state changed. Reason: New leader 0b6d9b67235e4dcca250ed38dbed9d4b. Latest consensus state: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:34.194391 2024 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:32:34.194535 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:32:34.194646 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:32:34.194633 1507 catalog_manager.cc:5495] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6d9b67235e4dcca250ed38dbed9d4b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260926 09:32:34.210973 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:34.208512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.0.98.3:41211)
0926 09:32:34.208561 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:34.208563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:34.208572 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:34.208638 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:34.208639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:34.208650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:34.208706 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:34.208865 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.209478 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.210241 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.210459 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.210535 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:34.210723 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:34.210758 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:34.210799 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:34.210868 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 09:32:34.210890 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:32:34.210976 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:34.208431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41211 (local address 127.0.0.1:52116)
0926 09:32:34.208508 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:34.208520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:34.208571 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:34.208714 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:34.208715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:34.208716 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:32:34.208785 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:34.208788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.209559 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.209560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:34.210153 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:34.210155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.210533 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.210534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:34.210604 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:34.210629 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:32:34.210810 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:32:34.210818 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 09:32:34.210855 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 09:32:34.210909 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:32:34.212021 1805 raft_consensus.cc:1261] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Refusing update from remote peer 0b6d9b67235e4dcca250ed38dbed9d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:34.212024 1665 raft_consensus.cc:1261] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Refusing update from remote peer 0b6d9b67235e4dcca250ed38dbed9d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:34.212260 2024 consensus_queue.cc:1035] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:34.212363 2032 consensus_queue.cc:1035] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:34.212864 2019 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: TxnStatusTablet state changed. Reason: New leader 0b6d9b67235e4dcca250ed38dbed9d4b. Latest consensus state: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:34.212955 2019 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:34.213020 2021 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: TxnStatusTablet state changed. Reason: New leader 0b6d9b67235e4dcca250ed38dbed9d4b. Latest consensus state: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:34.213128 2021 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:34.213845 2032 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:34.213928 2022 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:34.214143 2022 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:32:34.213938 2032 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0b6d9b67235e4dcca250ed38dbed9d4b: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:32:34.215147 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333540471650689024 to 7333540471579136000. Current Snapshot: MvccSnapshot[applied={T|T < 7333540471650689024}]
I20260926 09:32:34.215165 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333540471650689024 to 7333540471579136000. Current Snapshot: MvccSnapshot[applied={T|T < 7333540471650689024 or (T in {7333540471650689024})}]
I20260926 09:32:34.215279 2019 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:34.215376 2019 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 0cf1fd75a5c742aaab1053dc5bf61acd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:34.215409 2021 tablet_replica.cc:439] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 } } peers { permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 } } peers { permanent_uuid: "0b6d9b67235e4dcca250ed38dbed9d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41211 } } }
I20260926 09:32:34.215492 2021 tablet_replica.cc:442] T 3b2cad074a554a85bcba2cf0f71c97d4 P 72cdbea125ff4a57a8c991426bce0621: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:34.221966 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:34.219226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57377 (local address 127.0.98.3:41211)
0926 09:32:34.219426 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:34.219427 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:34.219545 (+ 118us) server_negotiation.cc:407] Connection header received
0926 09:32:34.219624 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:34.219624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:34.219632 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:34.219664 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:34.219834 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.220482 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.221194 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.221399 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.221496 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:34.221667 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:34.221700 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:34.221792 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:34.221853 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:32:34.221888 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":177}
I20260926 09:32:34.221979 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:34.219148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41211 (local address 127.0.98.3:57377)
0926 09:32:34.219488 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:34.219498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:34.219565 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:34.219723 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:34.219724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:34.219726 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:32:34.219770 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:34.219772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.220550 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.220550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:34.221109 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:34.221110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.221510 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.221511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:34.221565 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:34.221577 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:32:34.221795 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:32:34.221796 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:32:34.221849 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:32:34.221904 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260926 09:32:34.224109 1665 raft_consensus.cc:1261] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Refusing update from remote peer 0b6d9b67235e4dcca250ed38dbed9d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:34.224166 1805 raft_consensus.cc:1261] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Refusing update from remote peer 0b6d9b67235e4dcca250ed38dbed9d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:34.224375 2032 consensus_queue.cc:1035] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf1fd75a5c742aaab1053dc5bf61acd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:34.224502 2024 consensus_queue.cc:1035] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "72cdbea125ff4a57a8c991426bce0621" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:34.239363 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:34.239480 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:34.240139 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d87f0db179347fd8ad22dbcc4244e08: Sending DeleteTablet for 3 replicas of tablet fc1d0782e61841f0992c655380285a3f
I20260926 09:32:34.240423 1627 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1d0782e61841f0992c655380285a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57548
I20260926 09:32:34.240549 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1d0782e61841f0992c655380285a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52096
I20260926 09:32:34.240585 2039 tablet_replica.cc:330] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: stopping tablet replica
I20260926 09:32:34.240653 2038 tablet_replica.cc:330] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: stopping tablet replica
I20260926 09:32:34.240656 2039 raft_consensus.cc:2229] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:34.240695 2039 raft_consensus.cc:2258] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:34.240710 2038 raft_consensus.cc:2229] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:34.240784 2038 raft_consensus.cc:2258] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:34.240775 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1d0782e61841f0992c655380285a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36848
I20260926 09:32:34.240863 2037 tablet_replica.cc:330] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: stopping tablet replica
I20260926 09:32:34.240903 2037 raft_consensus.cc:2229] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:34.240936 2037 raft_consensus.cc:2258] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:34.241041 2039 ts_tablet_manager.cc:1905] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:34.241132 2038 ts_tablet_manager.cc:1905] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:34.242121 2037 ts_tablet_manager.cc:1905] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:34.242835 2039 ts_tablet_manager.cc:1918] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:34.242835 2038 ts_tablet_manager.cc:1918] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:34.242888 2038 log.cc:1196] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/fc1d0782e61841f0992c655380285a3f
I20260926 09:32:34.242888 2039 log.cc:1196] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/fc1d0782e61841f0992c655380285a3f
I20260926 09:32:34.243113 2039 ts_tablet_manager.cc:1939] T fc1d0782e61841f0992c655380285a3f P 0cf1fd75a5c742aaab1053dc5bf61acd: Deleting consensus metadata
I20260926 09:32:34.243113 2038 ts_tablet_manager.cc:1939] T fc1d0782e61841f0992c655380285a3f P 0b6d9b67235e4dcca250ed38dbed9d4b: Deleting consensus metadata
I20260926 09:32:34.243680 1495 catalog_manager.cc:4841] TS 0b6d9b67235e4dcca250ed38dbed9d4b (127.0.98.3:41211): tablet fc1d0782e61841f0992c655380285a3f (table test-table [id=f8770054727e41618de2d6d853932433]) successfully deleted
I20260926 09:32:34.243713 2037 ts_tablet_manager.cc:1918] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:34.243737 1494 catalog_manager.cc:4841] TS 0cf1fd75a5c742aaab1053dc5bf61acd (127.0.98.1:36877): tablet fc1d0782e61841f0992c655380285a3f (table test-table [id=f8770054727e41618de2d6d853932433]) successfully deleted
I20260926 09:32:34.243764 2037 log.cc:1196] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/fc1d0782e61841f0992c655380285a3f
I20260926 09:32:34.243968 2037 ts_tablet_manager.cc:1939] T fc1d0782e61841f0992c655380285a3f P 72cdbea125ff4a57a8c991426bce0621: Deleting consensus metadata
I20260926 09:32:34.244356 1495 catalog_manager.cc:4841] TS 72cdbea125ff4a57a8c991426bce0621 (127.0.98.2:44781): tablet fc1d0782e61841f0992c655380285a3f (table test-table [id=f8770054727e41618de2d6d853932433]) successfully deleted
Sep 26 09:32:34 dist-test-slave-21h8 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415153, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:34.256844 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:34.247715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.98.1:36877)
0926 09:32:34.247767 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:34.247770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:34.247905 (+ 135us) server_negotiation.cc:407] Connection header received
0926 09:32:34.247998 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:34.247999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:34.248011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:34.248076 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:34.248709 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.249256 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.250042 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.250259 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.250351 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:34.253538 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:34.253553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:34.253553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:34.253562 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:34.255517 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:34.255939 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:34.255944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:34.255945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:34.256021 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:34.256240 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:34.256240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:34.256241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:34.256404 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:34.256578 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:34.256633 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 09:32:34.256728 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 09:32:34.257195 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57562
I20260926 09:32:34.265024 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:34.258246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.0.98.62:42255)
0926 09:32:34.258302 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:34.258304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:34.258314 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:34.258381 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:34.258382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:34.258394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:34.258453 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:34.259008 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.259691 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.260414 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:34.260628 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:34.260733 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:34.262050 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:34.262052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:34.262052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:34.262061 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:34.263767 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:34.264162 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:34.264163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:34.264164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:34.264239 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:34.264452 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:34.264453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:34.264453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:34.264584 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:34.264701 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:34.264818 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 09:32:34.264908 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 09:32:34.265311 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51952
I20260926 09:32:34.266181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 1580
I20260926 09:32:34.274333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 1720
I20260926 09:32:34.280704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 1860
I20260926 09:32:34.286158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 1476
2026-09-26T09:32:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3940 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:34 dist-test-slave-21h8 krb5kdc[2071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:34 dist-test-slave-21h8 krb5kdc[2071](info): set up 2 sockets
Sep 26 09:32:34 dist-test-slave-21h8 krb5kdc[2071](info): commencing operation
krb5kdc: starting...
W20260926 09:32:35.359076 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:35 dist-test-slave-21h8 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:35.477969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46037
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:35.560182 2084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:35.560482 2084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:35.560528 2084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:35.561782 2084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:35.561872 2084 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:32:35.561894 2084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:35.561918 2084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:35.561945 2084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:35.561977 2084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:35.563666 2084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46037
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:35.564002 2084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:35.564258 2084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:35.569361 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:35.569458 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:35.569591 2092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:35.569671 2084 server_base.cc:1034] running on GCE node
I20260926 09:32:35.569939 2084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:35.570292 2084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:35.571466 2084 hybrid_clock.cc:648] HybridClock initialized: now 1790415155571460 us; error 37 us; skew 500 ppm
Sep 26 09:32:35 dist-test-slave-21h8 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:35.574198 2084 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:35.574698 2084 webserver.cc:460] Webserver started at http://127.0.98.62:38311/ using document root <none> and password file <none>
I20260926 09:32:35.574957 2084 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:35.575006 2084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:35.575161 2084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:35.576397 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "fe525a6cb08e43e1b2b11e4b20c065ca"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "4a95700bc6a641eaf626d308463363ee"
server_key_iv: "4dfb91abe83752a13929e0ed4a452174"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.576822 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe525a6cb08e43e1b2b11e4b20c065ca"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "4a95700bc6a641eaf626d308463363ee"
server_key_iv: "4dfb91abe83752a13929e0ed4a452174"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.578562 2084 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 09:32:35.579674 2099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:35.579890 2084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:35.580000 2084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "fe525a6cb08e43e1b2b11e4b20c065ca"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "4a95700bc6a641eaf626d308463363ee"
server_key_iv: "4dfb91abe83752a13929e0ed4a452174"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.580093 2084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:35.590238 2084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:35.593009 2084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:35.593209 2084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:35.598071 2084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46037
I20260926 09:32:35.598174 2161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46037 every 8 connection(s)
I20260926 09:32:35.598522 2084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:35.599256 2162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:35.601706 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: Bootstrap starting.
I20260926 09:32:35.602348 2162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:35.602648 2162 log.cc:824] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:35.602910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 2084
I20260926 09:32:35.602984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:35.603160 392 external_mini_cluster.cc:1428] Setting key 60bf5a21ec8c6bc0dc0cf9226c1949c4
I20260926 09:32:35.603506 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: No bootstrap required, opened a new log
I20260926 09:32:35.606356 2162 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe525a6cb08e43e1b2b11e4b20c065ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46037 } }
I20260926 09:32:35.606539 2162 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:35.606577 2162 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe525a6cb08e43e1b2b11e4b20c065ca, State: Initialized, Role: FOLLOWER
I20260926 09:32:35.606707 2162 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe525a6cb08e43e1b2b11e4b20c065ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46037 } }
I20260926 09:32:35.606781 2162 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:35.606818 2162 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:35.606860 2162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:35.607601 2162 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe525a6cb08e43e1b2b11e4b20c065ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46037 } }
I20260926 09:32:35.607725 2162 leader_election.cc:302] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe525a6cb08e43e1b2b11e4b20c065ca; no voters:
I20260926 09:32:35.607937 2162 leader_election.cc:288] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:35.607964 2167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:35.608090 2167 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [term 1 LEADER]: Becoming Leader. State: Replica: fe525a6cb08e43e1b2b11e4b20c065ca, State: Running, Role: LEADER
I20260926 09:32:35.608163 2167 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe525a6cb08e43e1b2b11e4b20c065ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46037 } }
I20260926 09:32:35.608217 2162 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:35.608487 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe525a6cb08e43e1b2b11e4b20c065ca. Latest consensus state: current_term: 1 leader_uuid: "fe525a6cb08e43e1b2b11e4b20c065ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe525a6cb08e43e1b2b11e4b20c065ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46037 } } }
I20260926 09:32:35.608472 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe525a6cb08e43e1b2b11e4b20c065ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe525a6cb08e43e1b2b11e4b20c065ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46037 } } }
I20260926 09:32:35.608608 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:35.608618 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca [sys.catalog]: This master's current role is: LEADER
W20260926 09:32:35.610126 2183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:32:35.610183 2183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:32:35.610234 2173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:35.610400 2173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:32:35 dist-test-slave-21h8 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:35.615164 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.603752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.0.98.62:46037)
0926 09:32:35.603934 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:35.603944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:35.604016 (+ 72us) server_negotiation.cc:407] Connection header received
0926 09:32:35.604881 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:35.604891 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:35.604959 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:35.605116 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:35.605685 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.607129 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.608031 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.608699 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.608821 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.611728 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:35.611732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:35.611733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:35.611744 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:35.613598 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.613984 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.613988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.613988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.614082 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.614295 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.614296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.614296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.614564 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:35.614732 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:35.614831 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 09:32:35.614974 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260926 09:32:35.615757 2173 catalog_manager.cc:1347] Generated new cluster ID: b7cf5aca604a48e2bb0015a43168d0d4
I20260926 09:32:35.615823 2173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:35.637130 2173 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:35.637801 2173 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:35.640799 2173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: Generated new TSK 0
I20260926 09:32:35.641047 2173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:32:35.670495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46037
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:35.754405 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:35.754635 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:35.754663 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:35.755934 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:35.755982 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:35.756004 2188 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:32:35.756037 2188 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:32:35.756072 2188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:35.757534 2188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:35.757807 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:35.758045 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:35.763144 2196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:35.763211 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:35.763240 2193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:35.763388 2188 server_base.cc:1034] running on GCE node
I20260926 09:32:35.763583 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:35.763782 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:35.764971 2188 hybrid_clock.cc:648] HybridClock initialized: now 1790415155764959 us; error 44 us; skew 500 ppm
Sep 26 09:32:35 dist-test-slave-21h8 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:35.767495 2188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:35.767941 2188 webserver.cc:460] Webserver started at http://127.0.98.1:40239/ using document root <none> and password file <none>
I20260926 09:32:35.768172 2188 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:35.768214 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:35.768363 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:35.769415 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "854f5d63cae2496eb66d4b8ca5ef94b3"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "d4cbcf7c426676e397340999390212eb"
server_key_iv: "32f039a5e705e31cddb1ade0d2eb2080"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.769774 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "854f5d63cae2496eb66d4b8ca5ef94b3"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "d4cbcf7c426676e397340999390212eb"
server_key_iv: "32f039a5e705e31cddb1ade0d2eb2080"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.771163 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:35.771991 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:35.772205 2188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:35.772276 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "854f5d63cae2496eb66d4b8ca5ef94b3"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "d4cbcf7c426676e397340999390212eb"
server_key_iv: "32f039a5e705e31cddb1ade0d2eb2080"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.772369 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:35.796792 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:35.798817 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:35.798980 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:35.799662 2188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:35.799731 2188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:35.799764 2188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:35.799782 2188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:32:35 dist-test-slave-21h8 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:35.807139 2188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34149
I20260926 09:32:35.807582 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:32:35.807235 2318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34149 every 8 connection(s)
I20260926 09:32:35.810225 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.800025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43623 (local address 127.0.98.62:46037)
0926 09:32:35.800074 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:35.800076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:35.800371 (+ 295us) server_negotiation.cc:407] Connection header received
0926 09:32:35.800920 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:35.800922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:35.800936 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:35.800982 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:35.801846 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.802736 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.804076 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.804312 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.804407 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.807011 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:35.807013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:35.807014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:35.807025 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:35.808884 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.809403 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.809404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.809405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.809459 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.809635 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.809635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.809635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.809738 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:35.809783 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:35.810072 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 09:32:35.810160 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:32:35.810372 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.800023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46037 (local address 127.0.98.1:43623)
0926 09:32:35.800232 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:35.800269 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:35.800809 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:35.801056 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:35.801059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:35.801425 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:35.801747 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:35.801754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.802831 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.802832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:35.803982 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:35.803985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.804431 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.804432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:35.804582 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.805231 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:35.805252 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:35.806864 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:35.809000 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:35.809006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:35.809008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:35.809318 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:35.809499 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:35.809499 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:35.809500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:35.809600 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:35.809777 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:35.809781 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:35.809916 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 09:32:35.810010 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 09:32:35.810146 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":135,"threads_started":1}
I20260926 09:32:35.810912 2319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46037
I20260926 09:32:35.811002 2319 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:35.811210 2319 heartbeater.cc:511] Master 127.0.98.62:46037 requested a full tablet report, sending...
I20260926 09:32:35.811761 2116 ts_manager.cc:194] Registered new tserver with Master: 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149)
I20260926 09:32:35.812489 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43623
I20260926 09:32:35.816161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 2188
I20260926 09:32:35.816269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:35.816483 392 external_mini_cluster.cc:1428] Setting key fee1e556684c5cc9bd1e23b3132838c1
I20260926 09:32:35.818320 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.812641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42269 (local address 127.0.98.62:46037)
0926 09:32:35.812704 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:35.812706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:35.812715 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:35.812735 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:35.812736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:35.812748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:35.812804 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:35.813171 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.813723 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.814429 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.814614 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.814664 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.815640 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:35.815641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:35.815641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:35.815647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:35.817218 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.817635 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.817635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.817636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.817689 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.817887 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.817887 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.817888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.817985 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:35.818034 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:35.818191 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:32:35.818253 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:32:35.818478 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.811960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46037 (local address 127.0.98.1:42269)
0926 09:32:35.812126 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:35.812140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:35.812193 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:35.812824 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:35.812825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:35.813036 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:35.813089 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:35.813091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.813820 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.813820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:35.814355 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:35.814357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.814676 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.814677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:35.814733 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.815203 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:35.815208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:35.815567 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:35.817298 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:35.817299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:35.817300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:35.817558 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:35.817752 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:35.817753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:35.817754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:35.817827 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:35.818046 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:35.818047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:35.818097 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 09:32:35.818176 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:32:35.818300 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:35.843407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46037
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:35.930243 2328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:35.930469 2328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:35.930495 2328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:35.931711 2328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:35.931756 2328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:35.931777 2328 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:32:35.931789 2328 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:32:35.931808 2328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:35.933251 2328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:35.933447 2328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:35.933631 2328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:35.938707 2336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:35.938822 2333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:35.938707 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:35.938949 2328 server_base.cc:1034] running on GCE node
I20260926 09:32:35.939163 2328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:35.939472 2328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:35.940622 2328 hybrid_clock.cc:648] HybridClock initialized: now 1790415155940610 us; error 29 us; skew 500 ppm
Sep 26 09:32:35 dist-test-slave-21h8 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:35.943238 2328 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:35.943730 2328 webserver.cc:460] Webserver started at http://127.0.98.2:45859/ using document root <none> and password file <none>
I20260926 09:32:35.943988 2328 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:35.944060 2328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:35.944214 2328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:35.945310 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "0bc698440b854714a9bc5571a29dbc14"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "0ac5b39697063a5f9e9a9547ecdedaf5"
server_key_iv: "65bc95eafd5a6d8e075adf858145314a"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.945693 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0bc698440b854714a9bc5571a29dbc14"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "0ac5b39697063a5f9e9a9547ecdedaf5"
server_key_iv: "65bc95eafd5a6d8e075adf858145314a"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.947139 2328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:35.948093 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:35.948338 2328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:35.948442 2328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "0bc698440b854714a9bc5571a29dbc14"
format_stamp: "Formatted at 2026-09-26 09:32:35 on dist-test-slave-21h8"
server_key: "0ac5b39697063a5f9e9a9547ecdedaf5"
server_key_iv: "65bc95eafd5a6d8e075adf858145314a"
server_key_version: "encryptionkey@0"
I20260926 09:32:35.948539 2328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:35.977528 2328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:35.980341 2328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:35.980520 2328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:35.981109 2328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:35.981190 2328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:35.981235 2328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:35.981262 2328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:35.988921 2328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38283
I20260926 09:32:35.989006 2458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38283 every 8 connection(s)
Sep 26 09:32:35 dist-test-slave-21h8 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:35.989362 2328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:35.992278 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.981642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39263 (local address 127.0.98.62:46037)
0926 09:32:35.981707 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:35.981710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:35.981959 (+ 249us) server_negotiation.cc:407] Connection header received
0926 09:32:35.982531 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:35.982533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:35.982547 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:35.982615 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:35.983706 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.984356 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.986149 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.986484 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.986558 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.989701 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:35.989703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:35.989704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:35.989713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:35.990972 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.991440 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.991441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.991441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.991493 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.991722 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.991722 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.991723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.991833 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:35.991895 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:35.992146 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 09:32:35.992212 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:32:35.992446 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.981637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46037 (local address 127.0.98.2:39263)
0926 09:32:35.981825 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:35.981863 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:35.982410 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:35.982745 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:35.982750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:35.983182 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:35.983568 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:35.983578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.984438 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.984440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:35.986023 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:35.986027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.986599 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.986600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:35.986783 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.987512 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:35.987525 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:35.989571 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:35.991046 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:35.991052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:35.991055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:35.991288 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:35.991553 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:35.991553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:35.991554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:35.991660 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:35.991900 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:35.991904 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:35.992026 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 09:32:35.992105 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:32:35.992237 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260926 09:32:35.992992 2459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46037
I20260926 09:32:35.993079 2459 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:35.993278 2459 heartbeater.cc:511] Master 127.0.98.62:46037 requested a full tablet report, sending...
I20260926 09:32:35.993719 2115 ts_manager.cc:194] Registered new tserver with Master: 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283)
I20260926 09:32:35.994194 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39263
I20260926 09:32:35.998795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 2328
I20260926 09:32:35.998906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:35.999111 392 external_mini_cluster.cc:1428] Setting key 20ef99bcbd2c1075b4b0bf6dc6f4f0df
I20260926 09:32:35.999601 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.994177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34083 (local address 127.0.98.62:46037)
0926 09:32:35.994235 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:35.994237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:35.994325 (+ 88us) server_negotiation.cc:407] Connection header received
0926 09:32:35.994396 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:35.994397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:35.994407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:35.994451 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:35.994774 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.995250 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.995984 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.996185 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.996260 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.997215 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:35.997216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:35.997217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:35.997224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:35.998398 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.998751 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.998752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.998753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.998827 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:35.999010 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:35.999011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:35.999012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:35.999196 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:35.999256 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:35.999433 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 09:32:35.999509 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:32:35.999619 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:35.994115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46037 (local address 127.0.98.2:34083)
0926 09:32:35.994263 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:35.994273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:35.994340 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:35.994463 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:35.994464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:35.994647 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:35.994699 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:35.994700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.995376 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.995377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:35.995893 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:35.995894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:35.996258 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:35.996259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:35.996346 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:35.996820 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:35.996825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:35.997108 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:35.998483 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:35.998484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:35.998485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:35.998679 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:35.998901 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:35.998901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:35.998901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:35.998961 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:35.999260 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:35.999262 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:35.999334 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 09:32:35.999408 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:32:35.999510 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:36.025933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46037
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:36.108807 2468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:36.109120 2468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:36.109164 2468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:36.110589 2468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:36.110663 2468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:36.110703 2468 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:32:36.110730 2468 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:32:36.110764 2468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:36.112457 2468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:36.112758 2468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:36.113004 2468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:32:36.118266 2468 server_base.cc:1034] running on GCE node
W20260926 09:32:36.118248 2476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:36.118234 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:36.118291 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:36.118695 2468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:36.118980 2468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:36.120191 2468 hybrid_clock.cc:648] HybridClock initialized: now 1790415156120172 us; error 34 us; skew 500 ppm
Sep 26 09:32:36 dist-test-slave-21h8 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:36.122725 2468 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:36.123265 2468 webserver.cc:460] Webserver started at http://127.0.98.3:41209/ using document root <none> and password file <none>
I20260926 09:32:36.123553 2468 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:36.123627 2468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:36.123788 2468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:36.124902 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f2e92915c684681838e8b99b223ef0a"
format_stamp: "Formatted at 2026-09-26 09:32:36 on dist-test-slave-21h8"
server_key: "ae9152ac4ae4ad6712220a44667c5535"
server_key_iv: "d0e0d8f8311e515ded6a51edcd43aeeb"
server_key_version: "encryptionkey@0"
I20260926 09:32:36.125296 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f2e92915c684681838e8b99b223ef0a"
format_stamp: "Formatted at 2026-09-26 09:32:36 on dist-test-slave-21h8"
server_key: "ae9152ac4ae4ad6712220a44667c5535"
server_key_iv: "d0e0d8f8311e515ded6a51edcd43aeeb"
server_key_version: "encryptionkey@0"
I20260926 09:32:36.126698 2468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:32:36.127497 2483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:36.127666 2468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:36.127787 2468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "6f2e92915c684681838e8b99b223ef0a"
format_stamp: "Formatted at 2026-09-26 09:32:36 on dist-test-slave-21h8"
server_key: "ae9152ac4ae4ad6712220a44667c5535"
server_key_iv: "d0e0d8f8311e515ded6a51edcd43aeeb"
server_key_version: "encryptionkey@0"
I20260926 09:32:36.127876 2468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:36.141610 2468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:36.144209 2468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:36.144388 2468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:36.144956 2468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:36.145020 2468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:36.145067 2468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:36.145095 2468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:36.152029 2468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44817
I20260926 09:32:36.152108 2598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44817 every 8 connection(s)
I20260926 09:32:36.152419 2468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:32:36 dist-test-slave-21h8 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:36.155522 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.145456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38435 (local address 127.0.98.62:46037)
0926 09:32:36.145526 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.145529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.145773 (+ 244us) server_negotiation.cc:407] Connection header received
0926 09:32:36.146356 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.146358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.146372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.146471 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:36.147435 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.148279 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.149939 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.150275 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.150368 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.153035 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:36.153036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:36.153036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:36.153043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:36.154288 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:36.154654 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:36.154655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:36.154655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:36.154706 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:36.154921 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:36.154922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:36.154922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:36.155023 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:36.155076 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.155307 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.155456 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:32:36.155659 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.145453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46037 (local address 127.0.98.3:38435)
0926 09:32:36.145632 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.145674 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.146204 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.146551 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.146556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.146950 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:36.147252 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.147259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.148434 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.148436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.149828 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.149831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.150381 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.150382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.150528 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.151197 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:36.151206 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:36.152941 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:36.154360 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:36.154365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:36.154367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:36.154588 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:36.154765 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:36.154765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:36.154766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:36.154868 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:36.155080 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:36.155083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:36.155199 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.155279 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.155465 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
I20260926 09:32:36.156193 2599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46037
I20260926 09:32:36.156281 2599 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:36.156495 2599 heartbeater.cc:511] Master 127.0.98.62:46037 requested a full tablet report, sending...
I20260926 09:32:36.156881 2116 ts_manager.cc:194] Registered new tserver with Master: 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817)
I20260926 09:32:36.157275 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38435
I20260926 09:32:36.160892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 2468
I20260926 09:32:36.160964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:36.161171 392 external_mini_cluster.cc:1428] Setting key 84bb788660ce874d3808206e4c567f1f
I20260926 09:32:36.162009 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:32:36.163219 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.157326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45403 (local address 127.0.98.62:46037)
0926 09:32:36.157367 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.157369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.157549 (+ 180us) server_negotiation.cc:407] Connection header received
0926 09:32:36.157617 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.157618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.157628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.157671 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:36.157998 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.158476 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.159164 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.159445 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.159518 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.160502 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:36.160503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:36.160503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:36.160510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:36.162109 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:36.162441 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:36.162442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:36.162443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:36.162504 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:36.162708 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:36.162709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:36.162710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:36.162822 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:36.162893 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.163049 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.163125 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:32:36.163231 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.157277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46037 (local address 127.0.98.3:45403)
0926 09:32:36.157484 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.157493 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.157562 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.157684 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.157685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.157870 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:36.157921 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.157922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.158568 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.158569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.159080 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.159081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.159516 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.159517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.159602 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.160103 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:36.160107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:36.160392 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:36.162188 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:36.162189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:36.162190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:36.162368 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:36.162575 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:36.162576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:36.162576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:36.162641 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:36.162894 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:36.162895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:36.162938 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.163023 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.163127 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
Sep 26 09:32:36 dist-test-slave-21h8 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:32:36 dist-test-slave-21h8 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:36.184852 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.177667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.0.98.62:46037)
0926 09:32:36.177739 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.177741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.177752 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:32:36.177786 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.177787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.177796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.177849 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:36.178401 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.178886 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.179662 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.179843 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.179899 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.182133 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:36.182135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:36.182135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:36.182142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:36.183695 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:36.184157 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:36.184159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:36.184159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:36.184224 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:36.184434 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:36.184434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:36.184434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:36.184537 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:36.184609 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.184695 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.184756 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:32:36.186551 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:36.188347 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:36.195408 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.191289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.0.98.1:34149)
0926 09:32:36.191844 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.191860 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.191867 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:32:36.191896 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.191897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.191909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.191997 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.192089 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.192767 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.194429 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.194922 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.195014 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.195078 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.195226 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.195267 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":122,"threads_started":1}
I20260926 09:32:36.195393 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.190791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44817 (local address 127.0.0.1:45986)
0926 09:32:36.190975 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.191018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.191087 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.191511 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.191515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.191517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.191585 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.191587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.193021 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.193022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.194155 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.194158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.195082 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.195083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.195151 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.195156 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.195189 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.195247 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":111,"thread_start_us":73,"threads_started":1}
I20260926 09:32:36.195681 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.190996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.0.98.2:38283)
0926 09:32:36.191486 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.191492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.191502 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:36.191541 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.191542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.191558 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.191788 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.191846 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.192952 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.194418 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.195110 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.195437 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.195510 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.195538 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.195584 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":77,"threads_started":1}
I20260926 09:32:36.195847 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.190976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.0.98.3:44817)
0926 09:32:36.191184 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.191189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.191199 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:36.191241 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.191243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.191259 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.191434 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.191656 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.192937 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.194243 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.194977 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.195623 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.195686 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.195711 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.195755 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260926 09:32:36.195400 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.191060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34149 (local address 127.0.0.1:43490)
0926 09:32:36.191706 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.191714 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.191755 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.191994 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.191995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.191997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.192032 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.192034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.192897 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.192899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.194341 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.194343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.195013 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.195014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.195088 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.195102 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.195207 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.195256 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":35,"thread_start_us":68,"threads_started":1}
I20260926 09:32:36.196216 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.190846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38283 (local address 127.0.0.1:35002)
0926 09:32:36.191261 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.191274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.191368 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.191653 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.191653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.191655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.191704 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.191705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.193100 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.193100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.194308 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.194311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.195197 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.195198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.195248 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.195251 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.195279 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.195336 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":36,"thread_start_us":113,"threads_started":1}
I20260926 09:32:36.196584 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 396d0239a9724d1fbb8c349e54a5a063 (DEFAULT_TABLE table=test-table [id=3d787847a0ee4f0780e7c12a779faa51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:36.196806 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 396d0239a9724d1fbb8c349e54a5a063 (DEFAULT_TABLE table=test-table [id=3d787847a0ee4f0780e7c12a779faa51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:36.196920 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396d0239a9724d1fbb8c349e54a5a063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.197148 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396d0239a9724d1fbb8c349e54a5a063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.199993 2622 tablet_bootstrap.cc:492] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Bootstrap starting.
I20260926 09:32:36.200435 2623 tablet_bootstrap.cc:492] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Bootstrap starting.
I20260926 09:32:36.201023 2622 tablet_bootstrap.cc:654] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.201054 2623 tablet_bootstrap.cc:654] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.200964 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 396d0239a9724d1fbb8c349e54a5a063 (DEFAULT_TABLE table=test-table [id=3d787847a0ee4f0780e7c12a779faa51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:36.201344 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396d0239a9724d1fbb8c349e54a5a063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.201356 2622 log.cc:824] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:36.201377 2623 log.cc:824] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:36.202126 2622 tablet_bootstrap.cc:492] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: No bootstrap required, opened a new log
I20260926 09:32:36.202198 2622 ts_tablet_manager.cc:1397] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:36.204290 2622 raft_consensus.cc:348] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.204499 2622 raft_consensus.cc:374] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.204535 2622 raft_consensus.cc:729] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2e92915c684681838e8b99b223ef0a, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.204662 2622 consensus_queue.cc:260] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.204964 2622 ts_tablet_manager.cc:1428] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:32:36.204991 2599 heartbeater.cc:503] Master 127.0.98.62:46037 was elected leader, sending a full tablet report...
I20260926 09:32:36.205443 2627 tablet_bootstrap.cc:492] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Bootstrap starting.
I20260926 09:32:36.205485 2623 tablet_bootstrap.cc:492] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: No bootstrap required, opened a new log
I20260926 09:32:36.205567 2623 ts_tablet_manager.cc:1397] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 09:32:36.206122 2627 tablet_bootstrap.cc:654] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.206467 2627 log.cc:824] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:36.207161 2627 tablet_bootstrap.cc:492] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: No bootstrap required, opened a new log
I20260926 09:32:36.207212 2627 ts_tablet_manager.cc:1397] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:36.207603 2623 raft_consensus.cc:348] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.207759 2623 raft_consensus.cc:374] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.207811 2623 raft_consensus.cc:729] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854f5d63cae2496eb66d4b8ca5ef94b3, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.207928 2623 consensus_queue.cc:260] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.208318 2623 ts_tablet_manager.cc:1428] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:32:36.208334 2319 heartbeater.cc:503] Master 127.0.98.62:46037 was elected leader, sending a full tablet report...
I20260926 09:32:36.209105 2627 raft_consensus.cc:348] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.209271 2627 raft_consensus.cc:374] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.209313 2627 raft_consensus.cc:729] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc698440b854714a9bc5571a29dbc14, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.209461 2627 consensus_queue.cc:260] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.209709 2627 ts_tablet_manager.cc:1428] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:32:36.209771 2459 heartbeater.cc:503] Master 127.0.98.62:46037 was elected leader, sending a full tablet report...
I20260926 09:32:36.212900 2630 raft_consensus.cc:482] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:36.212980 2630 raft_consensus.cc:504] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.213289 2630 leader_election.cc:288] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817), 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149)
I20260926 09:32:36.217150 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.213557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48439 (local address 127.0.98.3:44817)
0926 09:32:36.213644 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.213646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.213684 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:32:36.213741 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.213742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.213753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.213804 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.213938 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.214670 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.216208 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.216765 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.216844 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.216857 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.217036 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.217078 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 09:32:36.217150 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.213538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55421 (local address 127.0.98.1:34149)
0926 09:32:36.213582 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.213584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.213594 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:36.213663 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.213664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.213673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.213730 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.213917 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.214673 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.216208 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.216765 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.216856 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.216868 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.217036 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.217078 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:32:36.217170 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.213485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34149 (local address 127.0.98.2:55421)
0926 09:32:36.213541 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.213554 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.213592 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.213744 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.213746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.213747 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.213815 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.213816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.214762 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.214762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.216111 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.216115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.216862 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.216863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.216955 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.216957 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.217015 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.217067 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":51}
I20260926 09:32:36.217597 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396d0239a9724d1fbb8c349e54a5a063" candidate_uuid: "0bc698440b854714a9bc5571a29dbc14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" is_pre_election: true
I20260926 09:32:36.217180 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.213498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44817 (local address 127.0.98.2:48439)
0926 09:32:36.213635 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.213644 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.213691 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.213808 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.213808 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.213809 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.213873 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.213874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.214762 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.214762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.216111 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.216115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.216862 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.216863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.216946 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.216951 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.217015 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.217067 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":67,"thread_start_us":93,"threads_started":1}
I20260926 09:32:36.217772 2273 raft_consensus.cc:2454] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc698440b854714a9bc5571a29dbc14 in term 0.
I20260926 09:32:36.218046 2345 leader_election.cc:302] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc698440b854714a9bc5571a29dbc14, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:36.218039 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396d0239a9724d1fbb8c349e54a5a063" candidate_uuid: "0bc698440b854714a9bc5571a29dbc14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2e92915c684681838e8b99b223ef0a" is_pre_election: true
I20260926 09:32:36.218197 2553 raft_consensus.cc:2454] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc698440b854714a9bc5571a29dbc14 in term 0.
I20260926 09:32:36.218186 2630 raft_consensus.cc:2781] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:36.218237 2630 raft_consensus.cc:482] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:36.218271 2630 raft_consensus.cc:3037] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.219015 2630 raft_consensus.cc:504] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.219173 2630 leader_election.cc:288] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [CANDIDATE]: Term 1 election: Requested vote from peers 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817), 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149)
I20260926 09:32:36.219373 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396d0239a9724d1fbb8c349e54a5a063" candidate_uuid: "0bc698440b854714a9bc5571a29dbc14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3"
I20260926 09:32:36.219372 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396d0239a9724d1fbb8c349e54a5a063" candidate_uuid: "0bc698440b854714a9bc5571a29dbc14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2e92915c684681838e8b99b223ef0a"
I20260926 09:32:36.219449 2553 raft_consensus.cc:3037] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.219478 2273 raft_consensus.cc:3037] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.220074 2553 raft_consensus.cc:2454] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc698440b854714a9bc5571a29dbc14 in term 1.
I20260926 09:32:36.220201 2273 raft_consensus.cc:2454] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc698440b854714a9bc5571a29dbc14 in term 1.
I20260926 09:32:36.220570 2347 leader_election.cc:302] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc698440b854714a9bc5571a29dbc14, 6f2e92915c684681838e8b99b223ef0a; no voters:
I20260926 09:32:36.220685 2630 raft_consensus.cc:2781] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:36.220831 2630 raft_consensus.cc:686] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc698440b854714a9bc5571a29dbc14, State: Running, Role: LEADER
I20260926 09:32:36.220927 2630 consensus_queue.cc:237] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } }
I20260926 09:32:36.221781 2116 catalog_manager.cc:5495] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0bc698440b854714a9bc5571a29dbc14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } health_report { overall_health: UNKNOWN } } }
W20260926 09:32:36.239951 2460 tablet.cc:2369] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 09:32:36 dist-test-slave-21h8 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:36.255398 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.248067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46037 (local address 127.0.0.1:59928)
0926 09:32:36.248318 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.248340 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.248436 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.248603 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.248603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.248962 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:36.249013 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.249015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.249770 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.249771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.250388 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.250391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.250802 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.250803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.250861 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.251537 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:36.251543 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:36.253011 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:36.254395 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:36.254402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:36.254404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:36.254607 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:36.254779 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:36.254779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:36.254780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:36.254835 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:36.255057 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:36.255063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:36.255107 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.255158 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.255252 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":113,"threads_started":1}
I20260926 09:32:36.255448 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.248171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.0.98.62:46037)
0926 09:32:36.248282 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.248284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.248413 (+ 129us) server_negotiation.cc:407] Connection header received
0926 09:32:36.248516 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.248517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.248529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.248594 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:36.249110 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.249677 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.250482 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.250698 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.250793 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.253103 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:36.253105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:36.253105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:36.253111 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:36.254320 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:36.254673 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:36.254674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:36.254674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:36.254722 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:36.254898 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:36.254898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:36.254899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:36.255005 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:36.255053 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.255189 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.255300 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":78}
I20260926 09:32:36.257630 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59928:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:32:36.257825 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:36.259367 2253 tablet_service.cc:1467] Processing CreateTablet for tablet cf6b2b7ee9ac44e48f6319f6d4f67310 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56928bf3e8864533b2581ad37d51dbda]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:32:36.259346 2533 tablet_service.cc:1467] Processing CreateTablet for tablet cf6b2b7ee9ac44e48f6319f6d4f67310 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56928bf3e8864533b2581ad37d51dbda]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:32:36.259377 2393 tablet_service.cc:1467] Processing CreateTablet for tablet cf6b2b7ee9ac44e48f6319f6d4f67310 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56928bf3e8864533b2581ad37d51dbda]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:32:36.259532 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6b2b7ee9ac44e48f6319f6d4f67310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.259532 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6b2b7ee9ac44e48f6319f6d4f67310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.259546 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6b2b7ee9ac44e48f6319f6d4f67310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.260767 2627 tablet_bootstrap.cc:492] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: Bootstrap starting.
I20260926 09:32:36.261039 2623 tablet_bootstrap.cc:492] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: Bootstrap starting.
I20260926 09:32:36.261257 2627 tablet_bootstrap.cc:654] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.261780 2623 tablet_bootstrap.cc:654] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.262071 2627 tablet_bootstrap.cc:492] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: No bootstrap required, opened a new log
I20260926 09:32:36.262115 2627 ts_tablet_manager.cc:1397] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:36.262167 2622 tablet_bootstrap.cc:492] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: Bootstrap starting.
I20260926 09:32:36.262262 2627 raft_consensus.cc:348] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.262373 2627 raft_consensus.cc:374] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.262393 2627 raft_consensus.cc:729] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc698440b854714a9bc5571a29dbc14, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.262435 2623 tablet_bootstrap.cc:492] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: No bootstrap required, opened a new log
I20260926 09:32:36.262441 2627 consensus_queue.cc:260] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.262498 2623 ts_tablet_manager.cc:1397] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:36.262540 2627 ts_tablet_manager.cc:1428] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:36.262652 2630 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.262681 2623 raft_consensus.cc:348] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.262734 2622 tablet_bootstrap.cc:654] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.262732 2630 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:36.262753 2623 raft_consensus.cc:374] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.262768 2623 raft_consensus.cc:729] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854f5d63cae2496eb66d4b8ca5ef94b3, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.262825 2623 consensus_queue.cc:260] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.262923 2623 ts_tablet_manager.cc:1428] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:36.262974 2629 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.263033 2629 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:36.263693 2622 tablet_bootstrap.cc:492] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: No bootstrap required, opened a new log
I20260926 09:32:36.263738 2622 ts_tablet_manager.cc:1397] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:36.263888 2622 raft_consensus.cc:348] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.263962 2622 raft_consensus.cc:374] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.263976 2622 raft_consensus.cc:729] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2e92915c684681838e8b99b223ef0a, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.264041 2622 consensus_queue.cc:260] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.264133 2622 ts_tablet_manager.cc:1428] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:36.264199 2626 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.264302 2626 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 09:32:36.308413 2320 tablet.cc:2369] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:32:36.403129 2600 tablet.cc:2369] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:32:36.576193 2626 raft_consensus.cc:482] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:36.576316 2626 raft_consensus.cc:504] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.576707 2626 leader_election.cc:288] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149), 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283)
I20260926 09:32:36.579725 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.577030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51515 (local address 127.0.98.1:34149)
0926 09:32:36.577077 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.577079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.577174 (+ 95us) server_negotiation.cc:407] Connection header received
0926 09:32:36.577254 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.577255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.577264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.577347 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.577506 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.577991 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.578945 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.579303 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.579445 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.579456 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.579623 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.579659 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:32:36.579725 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.576933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34149 (local address 127.0.98.3:51515)
0926 09:32:36.577095 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.577108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.577191 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.577351 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.577352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.577354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.577423 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.577425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.578078 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.578079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.578842 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.578844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.579446 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.579447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.579536 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.579539 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.579604 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.579647 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 09:32:36.579953 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.577084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38283 (local address 127.0.98.3:60711)
0926 09:32:36.577303 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.577310 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.577377 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.577525 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.577526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.577527 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.577597 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.577599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.578252 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.578253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.579021 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.579023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.579655 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.579657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.579724 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.579727 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.579802 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.579846 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260926 09:32:36.580013 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b2b7ee9ac44e48f6319f6d4f67310" candidate_uuid: "6f2e92915c684681838e8b99b223ef0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" is_pre_election: true
I20260926 09:32:36.580123 2273 raft_consensus.cc:2454] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f2e92915c684681838e8b99b223ef0a in term 0.
I20260926 09:32:36.580283 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.577148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60711 (local address 127.0.98.2:38283)
0926 09:32:36.577198 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.577200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.577370 (+ 170us) server_negotiation.cc:407] Connection header received
0926 09:32:36.577431 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.577432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.577446 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.577522 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.577681 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.578165 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.579121 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.579563 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.579644 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.579655 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.580164 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.580229 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:32:36.580343 2485 leader_election.cc:302] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f2e92915c684681838e8b99b223ef0a, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:36.580518 2626 raft_consensus.cc:2781] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:36.580567 2626 raft_consensus.cc:482] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:36.580593 2626 raft_consensus.cc:3037] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.580713 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b2b7ee9ac44e48f6319f6d4f67310" candidate_uuid: "6f2e92915c684681838e8b99b223ef0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc698440b854714a9bc5571a29dbc14" is_pre_election: true
I20260926 09:32:36.580840 2413 raft_consensus.cc:2454] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f2e92915c684681838e8b99b223ef0a in term 0.
I20260926 09:32:36.581372 2626 raft_consensus.cc:504] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.581513 2626 leader_election.cc:288] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [CANDIDATE]: Term 1 election: Requested vote from peers 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149), 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283)
I20260926 09:32:36.581707 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b2b7ee9ac44e48f6319f6d4f67310" candidate_uuid: "6f2e92915c684681838e8b99b223ef0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3"
I20260926 09:32:36.581777 2273 raft_consensus.cc:3037] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.582406 2273 raft_consensus.cc:2454] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f2e92915c684681838e8b99b223ef0a in term 1.
I20260926 09:32:36.582548 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b2b7ee9ac44e48f6319f6d4f67310" candidate_uuid: "6f2e92915c684681838e8b99b223ef0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc698440b854714a9bc5571a29dbc14"
I20260926 09:32:36.582595 2485 leader_election.cc:302] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f2e92915c684681838e8b99b223ef0a, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:36.582629 2413 raft_consensus.cc:3037] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.582700 2626 raft_consensus.cc:2781] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:36.582880 2626 raft_consensus.cc:686] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [term 1 LEADER]: Becoming Leader. State: Replica: 6f2e92915c684681838e8b99b223ef0a, State: Running, Role: LEADER
I20260926 09:32:36.582970 2626 consensus_queue.cc:237] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.583391 2644 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: TxnStatusTablet state changed. Reason: New leader 6f2e92915c684681838e8b99b223ef0a. Latest consensus state: current_term: 1 leader_uuid: "6f2e92915c684681838e8b99b223ef0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.583468 2644 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:32:36.583563 2413 raft_consensus.cc:2454] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f2e92915c684681838e8b99b223ef0a in term 1.
I20260926 09:32:36.583686 2646 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:32:36.583837 2646 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:32:36.584218 2114 catalog_manager.cc:5495] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f2e92915c684681838e8b99b223ef0a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6f2e92915c684681838e8b99b223ef0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } health_report { overall_health: HEALTHY } } }
I20260926 09:32:36.618594 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.615845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.0.98.3:44817)
0926 09:32:36.615917 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.615919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.615928 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:36.615987 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.615988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.616005 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.616062 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:36.616216 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.616692 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.617814 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.618013 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.618070 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.618276 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.618376 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.618436 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.618513 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.618533 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:32:36.618649 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.615746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44817 (local address 127.0.0.1:45996)
0926 09:32:36.615836 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.615851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.615905 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.616092 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.616092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.616094 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:32:36.616149 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.616152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.616767 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.616767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.617713 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.617716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.618093 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.618094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.618162 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.618182 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.618445 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.618452 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.618508 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.618563 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 09:32:36.620699 2413 raft_consensus.cc:1261] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Refusing update from remote peer 6f2e92915c684681838e8b99b223ef0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:36.620738 2273 raft_consensus.cc:1261] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Refusing update from remote peer 6f2e92915c684681838e8b99b223ef0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:36.621035 2626 consensus_queue.cc:1035] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:36.621204 2644 consensus_queue.cc:1035] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:36.622370 2630 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: TxnStatusTablet state changed. Reason: New leader 6f2e92915c684681838e8b99b223ef0a. Latest consensus state: current_term: 1 leader_uuid: "6f2e92915c684681838e8b99b223ef0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.622460 2630 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:36.622478 2629 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: TxnStatusTablet state changed. Reason: New leader 6f2e92915c684681838e8b99b223ef0a. Latest consensus state: current_term: 1 leader_uuid: "6f2e92915c684681838e8b99b223ef0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.622565 2629 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:36.623858 2649 mvcc.cc:205] Tried to move back new op lower bound from 7333540481515450368 to 7333540481364287488. Current Snapshot: MvccSnapshot[applied={T|T < 7333540481515450368}]
I20260926 09:32:36.623945 2650 mvcc.cc:205] Tried to move back new op lower bound from 7333540481515450368 to 7333540481364287488. Current Snapshot: MvccSnapshot[applied={T|T < 7333540481515450368}]
I20260926 09:32:36.624038 2630 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f2e92915c684681838e8b99b223ef0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.624112 2630 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 0bc698440b854714a9bc5571a29dbc14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:36.624101 2644 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f2e92915c684681838e8b99b223ef0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.624217 2644 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:32:36.624296 2654 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f2e92915c684681838e8b99b223ef0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.624325 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333540481515450368 to 7333540481364287488. Current Snapshot: MvccSnapshot[applied={T|T < 7333540481515450368}]
I20260926 09:32:36.624372 2654 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 6f2e92915c684681838e8b99b223ef0a: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:32:36.624408 2629 tablet_replica.cc:439] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f2e92915c684681838e8b99b223ef0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } } }
I20260926 09:32:36.624469 2629 tablet_replica.cc:442] T cf6b2b7ee9ac44e48f6319f6d4f67310 P 854f5d63cae2496eb66d4b8ca5ef94b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:32:36.630096 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.627100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.0.98.2:38283)
0926 09:32:36.627141 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.627143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.627155 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:32:36.627207 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.627208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.627226 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.627367 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:36.627515 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.628308 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.629106 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.629418 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.629499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.629609 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.629741 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.629820 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.629971 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.629996 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:36.635115 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.632740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54277 (local address 127.0.98.3:44817)
0926 09:32:36.632804 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.632806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.632864 (+ 58us) server_negotiation.cc:407] Connection header received
0926 09:32:36.632928 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.632929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.632950 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.632998 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:36.633123 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.633594 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.634385 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.634596 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.634668 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.634866 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.634900 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.634954 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.635020 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.635039 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:32:36.635142 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.632664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44817 (local address 127.0.98.2:54277)
0926 09:32:36.632812 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.632823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.632879 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.632999 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.633000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.633001 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:32:36.633058 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.633060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.633683 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.633684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.634299 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.634300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.634681 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.634682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.634753 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.634765 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.634963 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.634964 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.635015 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.635059 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260926 09:32:36.637739 2273 raft_consensus.cc:1261] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc698440b854714a9bc5571a29dbc14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:36.637888 2553 raft_consensus.cc:1261] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Refusing update from remote peer 0bc698440b854714a9bc5571a29dbc14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:36.638028 2632 consensus_queue.cc:1035] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:36.638157 2630 consensus_queue.cc:1035] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:36.647118 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.644405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38283 (local address 127.0.98.3:47227)
0926 09:32:36.644476 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.644487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.644522 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.644659 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.644659 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.644661 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:32:36.644719 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.644721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.645624 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.645625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.646324 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.646327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.646692 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.646693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.646749 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.646766 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.646946 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.646948 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.647002 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.647050 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:32:36.647113 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.644457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47227 (local address 127.0.98.2:38283)
0926 09:32:36.644495 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.644497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.644509 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:32:36.644574 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.644574 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.644583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.644655 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:36.644784 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.645536 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.646394 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.646607 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.646692 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.646842 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.646877 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.646945 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.647006 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.647031 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:32:36.656776 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:36.656927 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:36.657763 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: Sending DeleteTablet for 3 replicas of tablet 396d0239a9724d1fbb8c349e54a5a063
I20260926 09:32:36.658102 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 396d0239a9724d1fbb8c349e54a5a063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45986
I20260926 09:32:36.658169 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 396d0239a9724d1fbb8c349e54a5a063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43490
I20260926 09:32:36.658361 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 396d0239a9724d1fbb8c349e54a5a063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35002
I20260926 09:32:36.658577 2670 tablet_replica.cc:330] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: stopping tablet replica
I20260926 09:32:36.658674 2670 raft_consensus.cc:2229] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:36.658777 2670 raft_consensus.cc:2258] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:36.659054 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:36.659217 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:36.659281 2670 ts_tablet_manager.cc:1905] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:36.660789 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 2e563888f29b407fb9d4f5a30d4d3cc0 (DEFAULT_TABLE table=test-table [id=06d577d696b34b9db8dee55ff54f847c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:36.660912 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e563888f29b407fb9d4f5a30d4d3cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.661247 2670 ts_tablet_manager.cc:1918] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:36.661309 2670 log.cc:1196] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/396d0239a9724d1fbb8c349e54a5a063
I20260926 09:32:36.661551 2670 ts_tablet_manager.cc:1939] T 396d0239a9724d1fbb8c349e54a5a063 P 0bc698440b854714a9bc5571a29dbc14: Deleting consensus metadata
I20260926 09:32:36.662029 2101 catalog_manager.cc:4841] TS 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283): tablet 396d0239a9724d1fbb8c349e54a5a063 (table test-table [id=3d787847a0ee4f0780e7c12a779faa51]) successfully deleted
I20260926 09:32:36.662412 2622 tablet_bootstrap.cc:492] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: Bootstrap starting.
I20260926 09:32:36.662580 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 2e563888f29b407fb9d4f5a30d4d3cc0 (DEFAULT_TABLE table=test-table [id=06d577d696b34b9db8dee55ff54f847c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:36.662690 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e563888f29b407fb9d4f5a30d4d3cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.662848 2622 tablet_bootstrap.cc:654] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.663488 2622 tablet_bootstrap.cc:492] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: No bootstrap required, opened a new log
I20260926 09:32:36.663534 2622 ts_tablet_manager.cc:1397] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:36.663681 2622 raft_consensus.cc:348] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.663743 2622 raft_consensus.cc:374] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.663761 2622 raft_consensus.cc:729] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2e92915c684681838e8b99b223ef0a, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.663818 2622 consensus_queue.cc:260] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.663923 2622 ts_tablet_manager.cc:1428] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:36.664022 2627 tablet_bootstrap.cc:492] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: Bootstrap starting.
I20260926 09:32:36.664062 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 2e563888f29b407fb9d4f5a30d4d3cc0 (DEFAULT_TABLE table=test-table [id=06d577d696b34b9db8dee55ff54f847c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:36.664175 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e563888f29b407fb9d4f5a30d4d3cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:36.664403 2627 tablet_bootstrap.cc:654] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.665246 2627 tablet_bootstrap.cc:492] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: No bootstrap required, opened a new log
I20260926 09:32:36.665287 2627 ts_tablet_manager.cc:1397] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:36.665416 2627 raft_consensus.cc:348] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.665485 2627 raft_consensus.cc:374] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.665512 2627 raft_consensus.cc:729] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc698440b854714a9bc5571a29dbc14, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.665558 2627 consensus_queue.cc:260] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.665617 2623 tablet_bootstrap.cc:492] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: Bootstrap starting.
I20260926 09:32:36.665652 2627 ts_tablet_manager.cc:1428] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:36.666050 2623 tablet_bootstrap.cc:654] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:36.666900 2669 tablet_replica.cc:330] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: stopping tablet replica
I20260926 09:32:36.667101 2669 raft_consensus.cc:2229] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:36.667215 2669 raft_consensus.cc:2258] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:36.667672 2623 tablet_bootstrap.cc:492] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: No bootstrap required, opened a new log
I20260926 09:32:36.667735 2623 ts_tablet_manager.cc:1397] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:36.667752 2669 ts_tablet_manager.cc:1905] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:36.667891 2623 raft_consensus.cc:348] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.667968 2623 raft_consensus.cc:374] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:36.667987 2623 raft_consensus.cc:729] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854f5d63cae2496eb66d4b8ca5ef94b3, State: Initialized, Role: FOLLOWER
I20260926 09:32:36.668030 2623 consensus_queue.cc:260] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.668118 2623 ts_tablet_manager.cc:1428] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:36.669237 2668 tablet_replica.cc:330] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: stopping tablet replica
I20260926 09:32:36.669384 2668 raft_consensus.cc:2229] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:36.669538 2669 ts_tablet_manager.cc:1918] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:36.669628 2669 log.cc:1196] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/396d0239a9724d1fbb8c349e54a5a063
I20260926 09:32:36.669878 2669 ts_tablet_manager.cc:1939] T 396d0239a9724d1fbb8c349e54a5a063 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting consensus metadata
I20260926 09:32:36.670285 2101 catalog_manager.cc:4841] TS 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149): tablet 396d0239a9724d1fbb8c349e54a5a063 (table test-table [id=3d787847a0ee4f0780e7c12a779faa51]) successfully deleted
I20260926 09:32:36.670346 2668 raft_consensus.cc:2258] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:36.670717 2668 ts_tablet_manager.cc:1905] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:36.672382 2668 ts_tablet_manager.cc:1918] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:36.672469 2668 log.cc:1196] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/396d0239a9724d1fbb8c349e54a5a063
I20260926 09:32:36.672761 2668 ts_tablet_manager.cc:1939] T 396d0239a9724d1fbb8c349e54a5a063 P 6f2e92915c684681838e8b99b223ef0a: Deleting consensus metadata
I20260926 09:32:36.673156 2103 catalog_manager.cc:4841] TS 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817): tablet 396d0239a9724d1fbb8c349e54a5a063 (table test-table [id=3d787847a0ee4f0780e7c12a779faa51]) successfully deleted
I20260926 09:32:36.694682 2629 raft_consensus.cc:482] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:36.694819 2629 raft_consensus.cc:504] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.695189 2629 leader_election.cc:288] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283), 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817)
I20260926 09:32:36.699630 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.695534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46727 (local address 127.0.98.3:44817)
0926 09:32:36.695578 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.695580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.695619 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:32:36.695729 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.695730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.695739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.695770 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.696033 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.696833 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.698587 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.699136 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.699181 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.699192 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.699535 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.699574 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:32:36.699613 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.695851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50275 (local address 127.0.98.2:38283)
0926 09:32:36.695894 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.695897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.695988 (+ 91us) server_negotiation.cc:407] Connection header received
0926 09:32:36.696084 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.696085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.696094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.696121 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:36.696326 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.696973 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.698588 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.699136 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.699181 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.699192 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.699491 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.699540 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:32:36.699889 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.695434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44817 (local address 127.0.98.1:46727)
0926 09:32:36.695557 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.695569 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.695650 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.695862 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.695863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.695865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.695950 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.695953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.697105 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.697105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.698476 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.698478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.699259 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.699260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.699454 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.699459 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.699706 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.699767 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":70,"thread_start_us":57,"threads_started":1}
I20260926 09:32:36.700153 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.695699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38283 (local address 127.0.98.1:50275)
0926 09:32:36.695936 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.695945 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.696011 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.696176 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.696177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.696179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:36.696237 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.696238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.697070 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.697071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.698475 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.698478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.699261 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.699261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.699393 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.699397 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.700035 (+ 638us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.700080 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":95,"thread_start_us":45,"threads_started":1}
I20260926 09:32:36.700160 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e563888f29b407fb9d4f5a30d4d3cc0" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2e92915c684681838e8b99b223ef0a" is_pre_election: true
I20260926 09:32:36.700255 2553 raft_consensus.cc:2454] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 0.
I20260926 09:32:36.700388 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e563888f29b407fb9d4f5a30d4d3cc0" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc698440b854714a9bc5571a29dbc14" is_pre_election: true
I20260926 09:32:36.700428 2207 leader_election.cc:302] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f2e92915c684681838e8b99b223ef0a, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:36.700491 2413 raft_consensus.cc:2454] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 0.
I20260926 09:32:36.700565 2629 raft_consensus.cc:2781] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:36.700604 2629 raft_consensus.cc:482] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:36.700623 2629 raft_consensus.cc:3037] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.701344 2629 raft_consensus.cc:504] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.701501 2629 leader_election.cc:288] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283), 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817)
I20260926 09:32:36.701682 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e563888f29b407fb9d4f5a30d4d3cc0" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc698440b854714a9bc5571a29dbc14"
I20260926 09:32:36.701696 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e563888f29b407fb9d4f5a30d4d3cc0" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2e92915c684681838e8b99b223ef0a"
I20260926 09:32:36.701750 2413 raft_consensus.cc:3037] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.701751 2553 raft_consensus.cc:3037] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:36.702533 2553 raft_consensus.cc:2454] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 1.
I20260926 09:32:36.702533 2413 raft_consensus.cc:2454] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 1.
I20260926 09:32:36.702703 2205 leader_election.cc:302] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc698440b854714a9bc5571a29dbc14, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:36.702813 2629 raft_consensus.cc:2781] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:36.702976 2629 raft_consensus.cc:686] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 LEADER]: Becoming Leader. State: Replica: 854f5d63cae2496eb66d4b8ca5ef94b3, State: Running, Role: LEADER
I20260926 09:32:36.703060 2629 consensus_queue.cc:237] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:36.703976 2115 catalog_manager.cc:5495] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } health_report { overall_health: UNKNOWN } } }
I20260926 09:32:36.719673 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.716965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.0.98.1:34149)
0926 09:32:36.717004 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.717006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.717037 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:32:36.717120 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.717120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.717141 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.717234 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:36.717504 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.718078 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.718833 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.719031 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.719104 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.719249 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.719429 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.719513 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.719555 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.719577 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:32:36.724761 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.722153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44817 (local address 127.0.98.1:55211)
0926 09:32:36.722316 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.722333 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.722404 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.722515 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.722515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.722517 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:32:36.722590 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.722591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.723205 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.723206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.723904 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.723907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.724295 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.724297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.724366 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.724379 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.724568 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.724570 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.724623 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.724669 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260926 09:32:36.724726 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.722204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55211 (local address 127.0.98.3:44817)
0926 09:32:36.722247 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.722250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.722389 (+ 139us) server_negotiation.cc:407] Connection header received
0926 09:32:36.722454 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.722455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.722466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.722515 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:36.722662 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.723123 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.723974 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.724183 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.724268 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.724462 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.724497 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.724567 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.724627 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.724645 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:36.727268 2413 raft_consensus.cc:1261] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Refusing update from remote peer 854f5d63cae2496eb66d4b8ca5ef94b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:36.727372 2553 raft_consensus.cc:1261] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Refusing update from remote peer 854f5d63cae2496eb66d4b8ca5ef94b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:36.727537 2629 consensus_queue.cc:1035] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:36.727759 2629 consensus_queue.cc:1035] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:36.736863 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.733825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49973 (local address 127.0.98.1:34149)
0926 09:32:36.733865 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:36.733867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:36.733875 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:32:36.733898 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:36.733899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:36.733909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:36.733957 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:36.734219 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.734905 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.736088 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.736383 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.736444 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.736614 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.736648 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.736700 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:36.736767 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 09:32:36.736789 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:32:36.736882 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:36.733605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34149 (local address 127.0.98.3:49973)
0926 09:32:36.733685 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:36.733698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:36.733738 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:36.734083 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:36.734084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:36.734086 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:32:36.734131 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.734133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.734988 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.734989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.735845 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:36.735848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:36.736475 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:36.736476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:36.736533 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:36.736540 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:32:36.736716 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:32:36.736718 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:32:36.736765 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:32:36.736815 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 09:32:36.737483 2651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 09:32:38.263680 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:38.263808 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:38.264761 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: Sending DeleteTablet for 3 replicas of tablet 2e563888f29b407fb9d4f5a30d4d3cc0
I20260926 09:32:38.265100 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e563888f29b407fb9d4f5a30d4d3cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43490
I20260926 09:32:38.265121 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e563888f29b407fb9d4f5a30d4d3cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35002
I20260926 09:32:38.265179 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e563888f29b407fb9d4f5a30d4d3cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45986
I20260926 09:32:38.265405 2709 tablet_replica.cc:330] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: stopping tablet replica
I20260926 09:32:38.265417 2708 tablet_replica.cc:330] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: stopping tablet replica
I20260926 09:32:38.265475 2709 raft_consensus.cc:2229] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:38.265479 2708 raft_consensus.cc:2229] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:38.265527 2709 raft_consensus.cc:2258] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:38.265585 2708 raft_consensus.cc:2258] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:38.265854 2709 ts_tablet_manager.cc:1905] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:38.267592 2708 ts_tablet_manager.cc:1905] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:38.268406 2707 tablet_replica.cc:330] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: stopping tablet replica
I20260926 09:32:38.268491 2707 raft_consensus.cc:2229] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:38.268582 2707 raft_consensus.cc:2258] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:38.268864 2707 ts_tablet_manager.cc:1905] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 09:32:38 dist-test-slave-21h8 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 09:32:38.299544 2707 ts_tablet_manager.cc:1918] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:32:38.299544 2709 ts_tablet_manager.cc:1918] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:32:38.299635 2709 log.cc:1196] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/2e563888f29b407fb9d4f5a30d4d3cc0
I20260926 09:32:38.299662 2708 ts_tablet_manager.cc:1918] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:32:38.299777 2708 log.cc:1196] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/2e563888f29b407fb9d4f5a30d4d3cc0
I20260926 09:32:38.299870 2709 ts_tablet_manager.cc:1939] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 6f2e92915c684681838e8b99b223ef0a: Deleting consensus metadata
I20260926 09:32:38.299896 2707 log.cc:1196] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/2e563888f29b407fb9d4f5a30d4d3cc0
I20260926 09:32:38.300024 2708 ts_tablet_manager.cc:1939] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting consensus metadata
I20260926 09:32:38.300259 2707 ts_tablet_manager.cc:1939] T 2e563888f29b407fb9d4f5a30d4d3cc0 P 0bc698440b854714a9bc5571a29dbc14: Deleting consensus metadata
Sep 26 09:32:38 dist-test-slave-21h8 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:38.314404 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:38.301316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.0.98.62:46037)
0926 09:32:38.301486 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:38.301488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:38.301527 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:32:38.301610 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:38.301612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:38.301628 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:38.301730 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:38.302321 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:38.303061 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:38.303870 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:38.304068 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:38.304174 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:38.311849 (+ 7675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:38.311851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:38.311851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:38.311861 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:38.313354 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:38.313690 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:38.313691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:38.313691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:38.313758 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:38.313953 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:38.313953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:38.313954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:38.314088 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:38.314166 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:38.314261 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 09:32:38.314329 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260926 09:32:38.315369 2101 catalog_manager.cc:4841] TS 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283): tablet 2e563888f29b407fb9d4f5a30d4d3cc0 (table test-table [id=06d577d696b34b9db8dee55ff54f847c]) successfully deleted
I20260926 09:32:38.315474 2101 catalog_manager.cc:4841] TS 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149): tablet 2e563888f29b407fb9d4f5a30d4d3cc0 (table test-table [id=06d577d696b34b9db8dee55ff54f847c]) successfully deleted
I20260926 09:32:38.315500 2103 catalog_manager.cc:4841] TS 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817): tablet 2e563888f29b407fb9d4f5a30d4d3cc0 (table test-table [id=06d577d696b34b9db8dee55ff54f847c]) successfully deleted
I20260926 09:32:38.316186 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:38.316347 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:38.318071 2253 tablet_service.cc:1467] Processing CreateTablet for tablet b49c5a4de0304dca89e2fa2ab2e27092 (DEFAULT_TABLE table=test-table [id=03c84e9323c448b5a0ae0140414c3fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:38.318120 2393 tablet_service.cc:1467] Processing CreateTablet for tablet b49c5a4de0304dca89e2fa2ab2e27092 (DEFAULT_TABLE table=test-table [id=03c84e9323c448b5a0ae0140414c3fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:38.318235 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49c5a4de0304dca89e2fa2ab2e27092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:38.318317 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49c5a4de0304dca89e2fa2ab2e27092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:38.319502 2718 tablet_bootstrap.cc:492] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: Bootstrap starting.
I20260926 09:32:38.319581 2719 tablet_bootstrap.cc:492] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: Bootstrap starting.
I20260926 09:32:38.319633 2532 tablet_service.cc:1467] Processing CreateTablet for tablet b49c5a4de0304dca89e2fa2ab2e27092 (DEFAULT_TABLE table=test-table [id=03c84e9323c448b5a0ae0140414c3fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:38.319763 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49c5a4de0304dca89e2fa2ab2e27092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:38.319949 2719 tablet_bootstrap.cc:654] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:38.320118 2718 tablet_bootstrap.cc:654] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:38.320583 2719 tablet_bootstrap.cc:492] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: No bootstrap required, opened a new log
I20260926 09:32:38.320657 2719 ts_tablet_manager.cc:1397] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:38.320783 2719 raft_consensus.cc:348] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.320866 2719 raft_consensus.cc:374] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:38.320899 2719 raft_consensus.cc:729] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc698440b854714a9bc5571a29dbc14, State: Initialized, Role: FOLLOWER
I20260926 09:32:38.320958 2719 consensus_queue.cc:260] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.321134 2719 ts_tablet_manager.cc:1428] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:38.321614 2718 tablet_bootstrap.cc:492] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: No bootstrap required, opened a new log
I20260926 09:32:38.321676 2718 ts_tablet_manager.cc:1397] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:38.321745 2723 tablet_bootstrap.cc:492] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: Bootstrap starting.
I20260926 09:32:38.321848 2718 raft_consensus.cc:348] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.321947 2718 raft_consensus.cc:374] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:38.321974 2718 raft_consensus.cc:729] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854f5d63cae2496eb66d4b8ca5ef94b3, State: Initialized, Role: FOLLOWER
I20260926 09:32:38.322021 2718 consensus_queue.cc:260] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.322147 2718 ts_tablet_manager.cc:1428] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:38.322302 2723 tablet_bootstrap.cc:654] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:38.323019 2723 tablet_bootstrap.cc:492] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: No bootstrap required, opened a new log
I20260926 09:32:38.323076 2723 ts_tablet_manager.cc:1397] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:38.323187 2723 raft_consensus.cc:348] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.323256 2723 raft_consensus.cc:374] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:38.323274 2723 raft_consensus.cc:729] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2e92915c684681838e8b99b223ef0a, State: Initialized, Role: FOLLOWER
I20260926 09:32:38.323366 2723 consensus_queue.cc:260] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.323508 2723 ts_tablet_manager.cc:1428] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:32:38.457602 2701 raft_consensus.cc:482] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:38.457720 2701 raft_consensus.cc:504] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.457912 2701 leader_election.cc:288] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283), 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817)
I20260926 09:32:38.458244 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49c5a4de0304dca89e2fa2ab2e27092" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2e92915c684681838e8b99b223ef0a" is_pre_election: true
I20260926 09:32:38.458240 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49c5a4de0304dca89e2fa2ab2e27092" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc698440b854714a9bc5571a29dbc14" is_pre_election: true
I20260926 09:32:38.458340 2553 raft_consensus.cc:2454] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 0.
I20260926 09:32:38.458365 2413 raft_consensus.cc:2454] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 0.
I20260926 09:32:38.458518 2207 leader_election.cc:302] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f2e92915c684681838e8b99b223ef0a, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:38.458642 2701 raft_consensus.cc:2781] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:38.458689 2701 raft_consensus.cc:482] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:38.458704 2701 raft_consensus.cc:3037] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:38.459456 2701 raft_consensus.cc:504] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.459610 2701 leader_election.cc:288] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283), 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817)
I20260926 09:32:38.459801 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49c5a4de0304dca89e2fa2ab2e27092" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2e92915c684681838e8b99b223ef0a"
I20260926 09:32:38.459769 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49c5a4de0304dca89e2fa2ab2e27092" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc698440b854714a9bc5571a29dbc14"
I20260926 09:32:38.459872 2553 raft_consensus.cc:3037] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:38.459893 2413 raft_consensus.cc:3037] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:38.460515 2553 raft_consensus.cc:2454] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 1.
I20260926 09:32:38.460520 2413 raft_consensus.cc:2454] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 1.
I20260926 09:32:38.460688 2207 leader_election.cc:302] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f2e92915c684681838e8b99b223ef0a, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:38.460788 2701 raft_consensus.cc:2781] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:38.460851 2701 raft_consensus.cc:686] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 LEADER]: Becoming Leader. State: Replica: 854f5d63cae2496eb66d4b8ca5ef94b3, State: Running, Role: LEADER
I20260926 09:32:38.460908 2701 consensus_queue.cc:237] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.461659 2115 catalog_manager.cc:5495] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } health_report { overall_health: UNKNOWN } } }
I20260926 09:32:38.469810 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:38.467586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.98.1:34149)
0926 09:32:38.467753 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:38.467756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:38.467764 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:32:38.467790 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:38.467790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:38.467802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:38.467859 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:38.467979 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:38.468476 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:38.469200 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:38.469408 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:38.469464 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:38.469586 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:38.469612 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:38.469644 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:38.469693 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 09:32:38.469709 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260926 09:32:38.472096 2553 raft_consensus.cc:1261] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Refusing update from remote peer 854f5d63cae2496eb66d4b8ca5ef94b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:38.472124 2413 raft_consensus.cc:1261] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Refusing update from remote peer 854f5d63cae2496eb66d4b8ca5ef94b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:38.472330 2685 consensus_queue.cc:1035] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:38.472461 2701 consensus_queue.cc:1035] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:38.482371 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:38.482547 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:38.483174 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: Sending DeleteTablet for 3 replicas of tablet b49c5a4de0304dca89e2fa2ab2e27092
I20260926 09:32:38.483418 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet b49c5a4de0304dca89e2fa2ab2e27092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43490
I20260926 09:32:38.483421 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet b49c5a4de0304dca89e2fa2ab2e27092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45986
I20260926 09:32:38.483465 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet b49c5a4de0304dca89e2fa2ab2e27092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35002
I20260926 09:32:38.483528 2708 tablet_replica.cc:330] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: stopping tablet replica
I20260926 09:32:38.483551 2709 tablet_replica.cc:330] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: stopping tablet replica
I20260926 09:32:38.483589 2708 raft_consensus.cc:2229] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:38.483611 2709 raft_consensus.cc:2229] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:38.483641 2708 raft_consensus.cc:2258] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:38.483664 2709 raft_consensus.cc:2258] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:38.483717 2707 tablet_replica.cc:330] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: stopping tablet replica
I20260926 09:32:38.483775 2707 raft_consensus.cc:2229] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:38.483814 2707 raft_consensus.cc:2258] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:38.484012 2709 ts_tablet_manager.cc:1905] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:38.484016 2708 ts_tablet_manager.cc:1905] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:38.484114 2707 ts_tablet_manager.cc:1905] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:38.485844 2708 ts_tablet_manager.cc:1918] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:38.485853 2709 ts_tablet_manager.cc:1918] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:38.485852 2707 ts_tablet_manager.cc:1918] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:32:38.485912 2708 log.cc:1196] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/b49c5a4de0304dca89e2fa2ab2e27092
I20260926 09:32:38.485909 2707 log.cc:1196] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/b49c5a4de0304dca89e2fa2ab2e27092
I20260926 09:32:38.485920 2709 log.cc:1196] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/b49c5a4de0304dca89e2fa2ab2e27092
I20260926 09:32:38.485965 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:38.486151 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:38.486160 2709 ts_tablet_manager.cc:1939] T b49c5a4de0304dca89e2fa2ab2e27092 P 6f2e92915c684681838e8b99b223ef0a: Deleting consensus metadata
I20260926 09:32:38.486161 2707 ts_tablet_manager.cc:1939] T b49c5a4de0304dca89e2fa2ab2e27092 P 0bc698440b854714a9bc5571a29dbc14: Deleting consensus metadata
I20260926 09:32:38.486162 2708 ts_tablet_manager.cc:1939] T b49c5a4de0304dca89e2fa2ab2e27092 P 854f5d63cae2496eb66d4b8ca5ef94b3: Deleting consensus metadata
I20260926 09:32:38.486508 2101 catalog_manager.cc:4841] TS 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149): tablet b49c5a4de0304dca89e2fa2ab2e27092 (table test-table [id=03c84e9323c448b5a0ae0140414c3fb3]) successfully deleted
I20260926 09:32:38.486581 2101 catalog_manager.cc:4841] TS 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283): tablet b49c5a4de0304dca89e2fa2ab2e27092 (table test-table [id=03c84e9323c448b5a0ae0140414c3fb3]) successfully deleted
I20260926 09:32:38.486678 2103 catalog_manager.cc:4841] TS 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817): tablet b49c5a4de0304dca89e2fa2ab2e27092 (table test-table [id=03c84e9323c448b5a0ae0140414c3fb3]) successfully deleted
I20260926 09:32:38.487622 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 1a973c2b27554caea891d1d7e3f2b8b7 (DEFAULT_TABLE table=test-table [id=9de9787e03c34b1ab4db2c6e9d05c0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:38.487622 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 1a973c2b27554caea891d1d7e3f2b8b7 (DEFAULT_TABLE table=test-table [id=9de9787e03c34b1ab4db2c6e9d05c0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:38.487658 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 1a973c2b27554caea891d1d7e3f2b8b7 (DEFAULT_TABLE table=test-table [id=9de9787e03c34b1ab4db2c6e9d05c0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:38.487739 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a973c2b27554caea891d1d7e3f2b8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:38.487748 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a973c2b27554caea891d1d7e3f2b8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:38.487778 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a973c2b27554caea891d1d7e3f2b8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:38.488960 2719 tablet_bootstrap.cc:492] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: Bootstrap starting.
I20260926 09:32:38.488979 2723 tablet_bootstrap.cc:492] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: Bootstrap starting.
I20260926 09:32:38.489296 2719 tablet_bootstrap.cc:654] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:38.489305 2723 tablet_bootstrap.cc:654] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:38.489609 2718 tablet_bootstrap.cc:492] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3: Bootstrap starting.
I20260926 09:32:38.489945 2718 tablet_bootstrap.cc:654] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:38.490007 2723 tablet_bootstrap.cc:492] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: No bootstrap required, opened a new log
I20260926 09:32:38.490048 2723 ts_tablet_manager.cc:1397] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:38.490144 2723 raft_consensus.cc:348] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.490264 2723 raft_consensus.cc:374] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:38.490293 2723 raft_consensus.cc:729] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2e92915c684681838e8b99b223ef0a, State: Initialized, Role: FOLLOWER
I20260926 09:32:38.490348 2719 tablet_bootstrap.cc:492] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: No bootstrap required, opened a new log
I20260926 09:32:38.490348 2723 consensus_queue.cc:260] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.490401 2719 ts_tablet_manager.cc:1397] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:38.490470 2723 ts_tablet_manager.cc:1428] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:38.490514 2719 raft_consensus.cc:348] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.490592 2719 raft_consensus.cc:374] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:38.490615 2719 raft_consensus.cc:729] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc698440b854714a9bc5571a29dbc14, State: Initialized, Role: FOLLOWER
I20260926 09:32:38.490664 2719 consensus_queue.cc:260] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.490758 2719 ts_tablet_manager.cc:1428] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:38.490999 2718 tablet_bootstrap.cc:492] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3: No bootstrap required, opened a new log
I20260926 09:32:38.491057 2718 ts_tablet_manager.cc:1397] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:38.491214 2718 raft_consensus.cc:348] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.491294 2718 raft_consensus.cc:374] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:38.491344 2718 raft_consensus.cc:729] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854f5d63cae2496eb66d4b8ca5ef94b3, State: Initialized, Role: FOLLOWER
I20260926 09:32:38.491389 2718 consensus_queue.cc:260] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.491482 2718 ts_tablet_manager.cc:1428] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:38.678210 2685 raft_consensus.cc:482] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:38.678361 2685 raft_consensus.cc:504] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.678691 2685 leader_election.cc:288] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283), 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817)
I20260926 09:32:38.678972 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a973c2b27554caea891d1d7e3f2b8b7" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc698440b854714a9bc5571a29dbc14" is_pre_election: true
I20260926 09:32:38.678972 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a973c2b27554caea891d1d7e3f2b8b7" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2e92915c684681838e8b99b223ef0a" is_pre_election: true
I20260926 09:32:38.679075 2413 raft_consensus.cc:2454] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 0.
I20260926 09:32:38.679075 2553 raft_consensus.cc:2454] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 0.
I20260926 09:32:38.679261 2205 leader_election.cc:302] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc698440b854714a9bc5571a29dbc14, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:38.679425 2685 raft_consensus.cc:2781] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:38.679494 2685 raft_consensus.cc:482] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:38.679509 2685 raft_consensus.cc:3037] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:38.680240 2685 raft_consensus.cc:504] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.680414 2685 leader_election.cc:288] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283), 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817)
I20260926 09:32:38.680593 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a973c2b27554caea891d1d7e3f2b8b7" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f2e92915c684681838e8b99b223ef0a"
I20260926 09:32:38.680682 2553 raft_consensus.cc:3037] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:38.680632 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a973c2b27554caea891d1d7e3f2b8b7" candidate_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc698440b854714a9bc5571a29dbc14"
I20260926 09:32:38.680742 2413 raft_consensus.cc:3037] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:38.681280 2553 raft_consensus.cc:2454] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 1.
I20260926 09:32:38.681389 2413 raft_consensus.cc:2454] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854f5d63cae2496eb66d4b8ca5ef94b3 in term 1.
I20260926 09:32:38.681493 2207 leader_election.cc:302] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f2e92915c684681838e8b99b223ef0a, 854f5d63cae2496eb66d4b8ca5ef94b3; no voters:
I20260926 09:32:38.681578 2685 raft_consensus.cc:2781] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:38.681659 2685 raft_consensus.cc:686] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [term 1 LEADER]: Becoming Leader. State: Replica: 854f5d63cae2496eb66d4b8ca5ef94b3, State: Running, Role: LEADER
I20260926 09:32:38.681716 2685 consensus_queue.cc:237] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } }
I20260926 09:32:38.682333 2115 catalog_manager.cc:5495] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854f5d63cae2496eb66d4b8ca5ef94b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 } health_report { overall_health: UNKNOWN } } }
I20260926 09:32:38.721345 2413 raft_consensus.cc:1261] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Refusing update from remote peer 854f5d63cae2496eb66d4b8ca5ef94b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:38.721391 2553 raft_consensus.cc:1261] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Refusing update from remote peer 854f5d63cae2496eb66d4b8ca5ef94b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:38.721621 2701 consensus_queue.cc:1035] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc698440b854714a9bc5571a29dbc14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:38.721722 2685 consensus_queue.cc:1035] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f2e92915c684681838e8b99b223ef0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:32:38.726135 2691 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 09:32:40.243922 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:40.244055 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:40.244989 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe525a6cb08e43e1b2b11e4b20c065ca: Sending DeleteTablet for 3 replicas of tablet 1a973c2b27554caea891d1d7e3f2b8b7
I20260926 09:32:40.245277 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a973c2b27554caea891d1d7e3f2b8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43490
I20260926 09:32:40.245260 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a973c2b27554caea891d1d7e3f2b8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45986
I20260926 09:32:40.245365 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a973c2b27554caea891d1d7e3f2b8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35002
I20260926 09:32:40.245600 2759 tablet_replica.cc:330] T 1a973c2b27554caea891d1d7e3f2b8b7 P 854f5d63cae2496eb66d4b8ca5ef94b3: stopping tablet replica
I20260926 09:32:40.245612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 2188
I20260926 09:32:40.246243 2760 tablet_replica.cc:330] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: stopping tablet replica
I20260926 09:32:40.246372 2760 raft_consensus.cc:2229] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:40.246484 2760 raft_consensus.cc:2258] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:40.246806 2760 ts_tablet_manager.cc:1905] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:40.245553 2761 tablet_replica.cc:330] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: stopping tablet replica
I20260926 09:32:40.247807 2761 raft_consensus.cc:2229] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:40.247896 2761 raft_consensus.cc:2258] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:40.248184 2761 ts_tablet_manager.cc:1905] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:40.248706 2760 ts_tablet_manager.cc:1918] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:32:40.248768 2760 log.cc:1196] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/1a973c2b27554caea891d1d7e3f2b8b7
I20260926 09:32:40.249013 2760 ts_tablet_manager.cc:1939] T 1a973c2b27554caea891d1d7e3f2b8b7 P 6f2e92915c684681838e8b99b223ef0a: Deleting consensus metadata
I20260926 09:32:40.249454 2103 catalog_manager.cc:4841] TS 6f2e92915c684681838e8b99b223ef0a (127.0.98.3:44817): tablet 1a973c2b27554caea891d1d7e3f2b8b7 (table test-table [id=9de9787e03c34b1ab4db2c6e9d05c0d5]) successfully deleted
I20260926 09:32:40.249814 2761 ts_tablet_manager.cc:1918] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:32:40.249864 2761 log.cc:1196] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/1a973c2b27554caea891d1d7e3f2b8b7
I20260926 09:32:40.250030 2761 ts_tablet_manager.cc:1939] T 1a973c2b27554caea891d1d7e3f2b8b7 P 0bc698440b854714a9bc5571a29dbc14: Deleting consensus metadata
I20260926 09:32:40.250406 2101 catalog_manager.cc:4841] TS 0bc698440b854714a9bc5571a29dbc14 (127.0.98.2:38283): tablet 1a973c2b27554caea891d1d7e3f2b8b7 (table test-table [id=9de9787e03c34b1ab4db2c6e9d05c0d5]) successfully deleted
W20260926 09:32:40.252053 2101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34149 (error 108)
I20260926 09:32:40.252298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 2328
I20260926 09:32:40.255534 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:40.252483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34149 (local address 127.0.0.1:43506)
0926 09:32:40.255437 (+ 2954us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:40.255494 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2918,"thread_start_us":80,"threads_started":1}
W20260926 09:32:40.255697 2101 catalog_manager.cc:4584] TS 854f5d63cae2496eb66d4b8ca5ef94b3 (127.0.98.1:34149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a973c2b27554caea891d1d7e3f2b8b7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34149: connect: Connection refused (error 111)
I20260926 09:32:40.258790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 2468
I20260926 09:32:40.265426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 2084
2026-09-26T09:32:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5980 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:40 dist-test-slave-21h8 krb5kdc[2766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:40 dist-test-slave-21h8 krb5kdc[2766](info): set up 2 sockets
Sep 26 09:32:40 dist-test-slave-21h8 krb5kdc[2766](info): commencing operation
krb5kdc: starting...
W20260926 09:32:41.303184 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:41 dist-test-slave-21h8 krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:41.425485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40399
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:41.508318 2779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:41.508541 2779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:41.508563 2779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:41.509758 2779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:41.509804 2779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:41.509819 2779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:41.509831 2779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:41.509845 2779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:41.511461 2779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:41.511727 2779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:41.511976 2779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:41.516829 2787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:41.516989 2779 server_base.cc:1034] running on GCE node
W20260926 09:32:41.516815 2785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:41.516819 2784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:41.517349 2779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:41.517611 2779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:41.518765 2779 hybrid_clock.cc:648] HybridClock initialized: now 1790415161518742 us; error 37 us; skew 500 ppm
Sep 26 09:32:41 dist-test-slave-21h8 krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:41.521456 2779 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:41.521850 2779 webserver.cc:460] Webserver started at http://127.0.98.62:38959/ using document root <none> and password file <none>
I20260926 09:32:41.522084 2779 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:41.522159 2779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:41.522315 2779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:41.523461 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "8a1c3ff3de7142ab99ff727f74995bd5"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "152b8d11a0cb87f3340d5aeb821108e3"
server_key_iv: "cfeabdc4c0fe3d461e7c7a546f3725dd"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.523829 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a1c3ff3de7142ab99ff727f74995bd5"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "152b8d11a0cb87f3340d5aeb821108e3"
server_key_iv: "cfeabdc4c0fe3d461e7c7a546f3725dd"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.525187 2779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:41.525951 2794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:41.526142 2779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:41.526254 2779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "8a1c3ff3de7142ab99ff727f74995bd5"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "152b8d11a0cb87f3340d5aeb821108e3"
server_key_iv: "cfeabdc4c0fe3d461e7c7a546f3725dd"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.526326 2779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:41.537866 2779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:41.540472 2779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:41.540778 2779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:41.544930 2779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40399
I20260926 09:32:41.544962 2846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40399 every 8 connection(s)
I20260926 09:32:41.545498 2779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:41.546233 2847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:41.548666 2847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5: Bootstrap starting.
I20260926 09:32:41.549356 2847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:41.549681 2847 log.cc:824] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:41.550541 2847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5: No bootstrap required, opened a new log
I20260926 09:32:41.550679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 2779
I20260926 09:32:41.550766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:41.550953 392 external_mini_cluster.cc:1428] Setting key 3f01a73b8ae1add91e2770c1a83b22c9
I20260926 09:32:41.553138 2847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1c3ff3de7142ab99ff727f74995bd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260926 09:32:41.553277 2847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:41.553299 2847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1c3ff3de7142ab99ff727f74995bd5, State: Initialized, Role: FOLLOWER
I20260926 09:32:41.553458 2847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1c3ff3de7142ab99ff727f74995bd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260926 09:32:41.553557 2847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:41.553601 2847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:41.553654 2847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:41.554219 2847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1c3ff3de7142ab99ff727f74995bd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260926 09:32:41.554334 2847 leader_election.cc:302] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a1c3ff3de7142ab99ff727f74995bd5; no voters:
I20260926 09:32:41.554525 2847 leader_election.cc:288] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:41.554561 2852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:41.554661 2852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [term 1 LEADER]: Becoming Leader. State: Replica: 8a1c3ff3de7142ab99ff727f74995bd5, State: Running, Role: LEADER
I20260926 09:32:41.554793 2852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1c3ff3de7142ab99ff727f74995bd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260926 09:32:41.554796 2847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:41.556627 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a1c3ff3de7142ab99ff727f74995bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1c3ff3de7142ab99ff727f74995bd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } } }
W20260926 09:32:41.556757 2867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 09:32:41.556757 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a1c3ff3de7142ab99ff727f74995bd5. Latest consensus state: current_term: 1 leader_uuid: "8a1c3ff3de7142ab99ff727f74995bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1c3ff3de7142ab99ff727f74995bd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } } }
W20260926 09:32:41.556919 2867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:32:41.556936 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:41.556805 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:41.557132 2869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:41.557345 2869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:41.558957 2869 catalog_manager.cc:1347] Generated new cluster ID: 850d4e8a436840b394364a16cef51ac4
I20260926 09:32:41.559008 2869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:32:41 dist-test-slave-21h8 krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:41.564291 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:41.551476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.0.98.62:40399)
0926 09:32:41.551673 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:41.551686 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:41.551715 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:32:41.553145 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:41.553154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:41.553210 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:41.553397 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:41.554152 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.555559 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.556883 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.557552 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.557665 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:41.560533 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:41.560537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:41.560538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:41.560549 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:41.562596 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:41.563013 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:41.563027 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:41.563027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:41.563148 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:41.563390 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:41.563391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:41.563391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:41.563690 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:41.563844 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:41.563951 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 09:32:41.564091 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260926 09:32:41.566430 2869 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:41.567108 2869 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:41.572928 2869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a1c3ff3de7142ab99ff727f74995bd5: Generated new TSK 0
I20260926 09:32:41.573132 2869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:32:41.602619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:41.687337 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:41.687572 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:41.687604 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:41.689002 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:41.689060 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:41.689100 2873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:41.690791 2873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:41.691097 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:41.691406 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:41.696779 2881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:41.696810 2879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:41.696779 2878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:41.697113 2873 server_base.cc:1034] running on GCE node
I20260926 09:32:41.697378 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:41.697672 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:41.698839 2873 hybrid_clock.cc:648] HybridClock initialized: now 1790415161698835 us; error 43 us; skew 500 ppm
Sep 26 09:32:41 dist-test-slave-21h8 krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:41.701777 2873 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:41.702325 2873 webserver.cc:460] Webserver started at http://127.0.98.1:37337/ using document root <none> and password file <none>
I20260926 09:32:41.702534 2873 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:41.702581 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:41.702685 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:41.703886 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "65d4d5a1c45247e9ab3fc20b1dc5f702"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "606d4eba643504e6ff8eec763d6ac382"
server_key_iv: "35de86ffc45d285374e57cd043bcc7bf"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.704236 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "65d4d5a1c45247e9ab3fc20b1dc5f702"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "606d4eba643504e6ff8eec763d6ac382"
server_key_iv: "35de86ffc45d285374e57cd043bcc7bf"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.705662 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:41.706502 2888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:41.706746 2873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:41.706835 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "65d4d5a1c45247e9ab3fc20b1dc5f702"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "606d4eba643504e6ff8eec763d6ac382"
server_key_iv: "35de86ffc45d285374e57cd043bcc7bf"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.706885 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:41.732270 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:41.734279 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:41.734396 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:41.734596 2873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:41.734946 2873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:41.734983 2873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:41.735006 2873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:41.735052 2873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:41.740877 2873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40139
I20260926 09:32:41.740957 3001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40139 every 8 connection(s)
I20260926 09:32:41.741314 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:32:41 dist-test-slave-21h8 krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:41.748083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 2873
I20260926 09:32:41.748206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:41.748407 392 external_mini_cluster.cc:1428] Setting key 4a4764904e1f2eccd5a4c65c1740e9a8
I20260926 09:32:41.753232 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:41.742170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60899 (local address 127.0.98.62:40399)
0926 09:32:41.742246 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:41.742248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:41.742452 (+ 204us) server_negotiation.cc:407] Connection header received
0926 09:32:41.743467 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:41.743468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:41.743479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:41.743543 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:41.744259 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.744788 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.745841 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.746071 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.746142 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:41.748554 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:41.748556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:41.748557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:41.748572 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:41.751491 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:41.752095 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:41.752096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:41.752096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:41.752157 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:41.752479 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:41.752480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:41.752480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:41.752594 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:41.752647 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:41.753072 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 09:32:41.753148 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:32:41.753491 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:41.742143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40399 (local address 127.0.98.1:60899)
0926 09:32:41.742314 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:41.742358 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:41.743363 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:41.743608 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:41.743611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:41.743937 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:41.744138 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:41.744143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.744886 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.744887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:41.745736 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:41.745739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.746160 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.746161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:41.746280 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:41.746900 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:41.746909 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:41.748457 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:41.751606 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:41.751618 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:41.751622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:41.752008 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:41.752245 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:41.752246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:41.752246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:41.752413 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:41.752660 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:41.752667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:41.752863 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 09:32:41.752995 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 09:32:41.753207 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260926 09:32:41.754148 3002 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40399
I20260926 09:32:41.754252 3002 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:41.754459 3002 heartbeater.cc:511] Master 127.0.98.62:40399 requested a full tablet report, sending...
I20260926 09:32:41.755093 2811 ts_manager.cc:194] Registered new tserver with Master: 65d4d5a1c45247e9ab3fc20b1dc5f702 (127.0.98.1:40139)
I20260926 09:32:41.755859 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60899
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:41.778738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:41.873531 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:41.873771 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:41.873799 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:41.875142 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:41.875197 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:41.875228 3007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:41.876953 3007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:41.877255 3007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:41.877521 3007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:41.883106 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:41.883137 3007 server_base.cc:1034] running on GCE node
W20260926 09:32:41.883095 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:41.883095 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:41.883553 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:41.883862 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:41.885046 3007 hybrid_clock.cc:648] HybridClock initialized: now 1790415161885030 us; error 35 us; skew 500 ppm
Sep 26 09:32:41 dist-test-slave-21h8 krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:41.887907 3007 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:41.888365 3007 webserver.cc:460] Webserver started at http://127.0.98.2:34249/ using document root <none> and password file <none>
I20260926 09:32:41.888576 3007 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:41.888618 3007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:41.888716 3007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:41.889863 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "4ea31ada05e346fa9e841ef9c5c53d73"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "029c33dddd1474db92b2c6a19f1c1402"
server_key_iv: "ba9139369d381ccd44f593440cfab461"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.890249 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4ea31ada05e346fa9e841ef9c5c53d73"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "029c33dddd1474db92b2c6a19f1c1402"
server_key_iv: "ba9139369d381ccd44f593440cfab461"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.891770 3007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:32:41.892589 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:41.892773 3007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:32:41.892890 3007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "4ea31ada05e346fa9e841ef9c5c53d73"
format_stamp: "Formatted at 2026-09-26 09:32:41 on dist-test-slave-21h8"
server_key: "029c33dddd1474db92b2c6a19f1c1402"
server_key_iv: "ba9139369d381ccd44f593440cfab461"
server_key_version: "encryptionkey@0"
I20260926 09:32:41.892979 3007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:41.907368 3007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:41.909382 3007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:41.909557 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:41.909816 3007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:41.910233 3007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:41.910269 3007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:41.910293 3007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:41.910344 3007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:41.916448 3007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44743
I20260926 09:32:41.916529 3135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44743 every 8 connection(s)
I20260926 09:32:41.916944 3007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:32:41 dist-test-slave-21h8 krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:41.925868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 3007
I20260926 09:32:41.925971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:41.926178 392 external_mini_cluster.cc:1428] Setting key 28b619f7f73e5ef1b898ec8bb5363e28
I20260926 09:32:41.927901 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:41.918068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40913 (local address 127.0.98.62:40399)
0926 09:32:41.918123 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:41.918126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:41.918172 (+ 46us) server_negotiation.cc:407] Connection header received
0926 09:32:41.919410 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:41.919411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:41.919426 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:41.919490 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:41.920416 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.921262 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.922311 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.922517 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.922574 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:41.924782 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:41.924783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:41.924783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:41.924790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:41.926355 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:41.926813 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:41.926815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:41.926816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:41.926888 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:41.927193 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:41.927194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:41.927194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:41.927368 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:41.927426 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:41.927723 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 09:32:41.927807 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:32:41.928144 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:41.917773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40399 (local address 127.0.98.2:40913)
0926 09:32:41.917993 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:41.918073 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:41.919266 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:41.919568 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:41.919572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:41.919973 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:41.920267 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:41.920274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.921375 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.921376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:41.922220 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:41.922222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:41.922592 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:41.922593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:41.922709 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:41.923186 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:41.923194 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:41.924682 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:41.926444 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:41.926451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:41.926454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:41.926721 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:41.926958 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:41.926959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:41.926960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:41.927110 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:41.927427 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:41.927435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:41.927585 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 09:32:41.927691 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:32:41.927864 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260926 09:32:41.928668 3136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40399
I20260926 09:32:41.928790 3136 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:41.928989 3136 heartbeater.cc:511] Master 127.0.98.62:40399 requested a full tablet report, sending...
I20260926 09:32:41.929399 2811 ts_manager.cc:194] Registered new tserver with Master: 4ea31ada05e346fa9e841ef9c5c53d73 (127.0.98.2:44743)
I20260926 09:32:41.929809 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40913
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:41.954017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:42.037873 3141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:42.038100 3141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:42.038125 3141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:42.039384 3141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:42.039428 3141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:42.039456 3141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:42.041033 3141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:42.041303 3141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:42.041579 3141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:42.046738 3146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:42.046849 3149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:42.046875 3147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:42.047071 3141 server_base.cc:1034] running on GCE node
I20260926 09:32:42.047375 3141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:42.047652 3141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:42.048806 3141 hybrid_clock.cc:648] HybridClock initialized: now 1790415162048791 us; error 21 us; skew 500 ppm
Sep 26 09:32:42 dist-test-slave-21h8 krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:42.051529 3141 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:42.052086 3141 webserver.cc:460] Webserver started at http://127.0.98.3:34227/ using document root <none> and password file <none>
I20260926 09:32:42.052356 3141 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:42.052433 3141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:42.052592 3141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:42.053761 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "acb7e6844d664cfab8c96c8841b2141f"
format_stamp: "Formatted at 2026-09-26 09:32:42 on dist-test-slave-21h8"
server_key: "a8381b8a969d6e3dfc957324da59b97e"
server_key_iv: "5e00e6d210785156895fd10ed5db847a"
server_key_version: "encryptionkey@0"
I20260926 09:32:42.054163 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "acb7e6844d664cfab8c96c8841b2141f"
format_stamp: "Formatted at 2026-09-26 09:32:42 on dist-test-slave-21h8"
server_key: "a8381b8a969d6e3dfc957324da59b97e"
server_key_iv: "5e00e6d210785156895fd10ed5db847a"
server_key_version: "encryptionkey@0"
I20260926 09:32:42.056003 3141 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 09:32:42.056927 3156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:42.057142 3141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:42.057251 3141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "acb7e6844d664cfab8c96c8841b2141f"
format_stamp: "Formatted at 2026-09-26 09:32:42 on dist-test-slave-21h8"
server_key: "a8381b8a969d6e3dfc957324da59b97e"
server_key_iv: "5e00e6d210785156895fd10ed5db847a"
server_key_version: "encryptionkey@0"
I20260926 09:32:42.057350 3141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:42.078122 3141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:42.080441 3141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:42.080636 3141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:42.080912 3141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:42.081275 3141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:42.081346 3141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:42.081394 3141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:42.081430 3141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:42.088474 3141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46441
I20260926 09:32:42.088533 3269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46441 every 8 connection(s)
I20260926 09:32:42.088929 3141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:32:42.089560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 3141
I20260926 09:32:42.089658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:42.089857 392 external_mini_cluster.cc:1428] Setting key 821231a0bcb74417d6bf590ef0739354
Sep 26 09:32:42 dist-test-slave-21h8 krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:42.099828 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:42.090107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57939 (local address 127.0.98.62:40399)
0926 09:32:42.090149 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:42.090151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:42.090160 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:42.091282 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:42.091283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:42.091297 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:42.091398 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:42.092130 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:42.092645 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:42.093859 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:42.094055 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:42.094132 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:42.096582 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:42.096584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:42.096584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:42.096593 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:42.098104 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:42.098546 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:42.098547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:42.098548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:42.098620 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:42.098989 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:42.098991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:42.098991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:42.099165 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:42.099260 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:42.099622 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 09:32:42.099718 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:42.100188 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:42.089802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40399 (local address 127.0.98.3:57939)
0926 09:32:42.089965 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:42.090010 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:42.091160 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:42.091485 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:42.091494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:42.091803 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:42.092016 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:42.092020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:42.092769 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:42.092769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:42.093750 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:42.093753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:42.094140 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:42.094141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:42.094274 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:42.094850 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:42.094859 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:42.096471 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:42.098196 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:42.098203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:42.098206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:42.098463 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:42.098689 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:42.098689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:42.098690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:42.098842 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:42.099251 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:42.099258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:42.099465 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 09:32:42.099599 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 09:32:42.099807 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260926 09:32:42.100723 3270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40399
I20260926 09:32:42.100828 3270 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:42.101008 3270 heartbeater.cc:511] Master 127.0.98.62:40399 requested a full tablet report, sending...
I20260926 09:32:42.101409 2811 ts_manager.cc:194] Registered new tserver with Master: acb7e6844d664cfab8c96c8841b2141f (127.0.98.3:46441)
I20260926 09:32:42.101876 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57939
I20260926 09:32:42.101903 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 09:32:42.114370 2851 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:32:42.113233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.0.98.62:40399)
0926 09:32:42.113301 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:42.113303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:42.113419 (+ 116us) server_negotiation.cc:407] Connection header received
0926 09:32:42.113468 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:42.113469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:42.113479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:42.113546 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:42.114270 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46808: BlockingRecv error: recv got EOF from 127.0.0.1:46808 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:32:42.115263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 2873
I20260926 09:32:42.120147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 3007
I20260926 09:32:42.124780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 3141
I20260926 09:32:42.129642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 2779
2026-09-26T09:32:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1862 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:42 dist-test-slave-21h8 krb5kdc[3282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:42 dist-test-slave-21h8 krb5kdc[3282](info): set up 2 sockets
Sep 26 09:32:42 dist-test-slave-21h8 krb5kdc[3282](info): commencing operation
krb5kdc: starting...
W20260926 09:32:43.179200 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:43 dist-test-slave-21h8 krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:43.303397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:43.386631 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:43.386852 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:43.386875 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:43.388195 3295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:43.388240 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:43.388255 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:43.388268 3295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:43.388280 3295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:43.389797 3295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:43.390069 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:43.390303 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:32:43.395514 3295 server_base.cc:1034] running on GCE node
W20260926 09:32:43.395476 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:43.395486 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:43.395496 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:43.396051 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:43.396363 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:43.397542 3295 hybrid_clock.cc:648] HybridClock initialized: now 1790415163397523 us; error 39 us; skew 500 ppm
Sep 26 09:32:43 dist-test-slave-21h8 krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:43.404382 3295 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:43.404940 3295 webserver.cc:460] Webserver started at http://127.0.98.62:36669/ using document root <none> and password file <none>
I20260926 09:32:43.405189 3295 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:43.405239 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:43.405378 3295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:43.406550 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "cf28fdf4de4649dd8bb5c433c1067e70"
format_stamp: "Formatted at 2026-09-26 09:32:43 on dist-test-slave-21h8"
server_key: "f21916ed701ba80e32eeb23146a03a29"
server_key_iv: "5e96e1c547366dccd6ec392066561729"
server_key_version: "encryptionkey@0"
I20260926 09:32:43.406941 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf28fdf4de4649dd8bb5c433c1067e70"
format_stamp: "Formatted at 2026-09-26 09:32:43 on dist-test-slave-21h8"
server_key: "f21916ed701ba80e32eeb23146a03a29"
server_key_iv: "5e96e1c547366dccd6ec392066561729"
server_key_version: "encryptionkey@0"
I20260926 09:32:43.408520 3295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:43.409399 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:43.409610 3295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:43.409713 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "cf28fdf4de4649dd8bb5c433c1067e70"
format_stamp: "Formatted at 2026-09-26 09:32:43 on dist-test-slave-21h8"
server_key: "f21916ed701ba80e32eeb23146a03a29"
server_key_iv: "5e96e1c547366dccd6ec392066561729"
server_key_version: "encryptionkey@0"
I20260926 09:32:43.409792 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:43.421567 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:43.424404 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:43.424603 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:43.429379 3295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41769
I20260926 09:32:43.429445 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41769 every 8 connection(s)
I20260926 09:32:43.429860 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:43.430955 3363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:43.435065 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70: Bootstrap starting.
I20260926 09:32:43.435925 3363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:43.436268 3363 log.cc:824] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:43.437096 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70: No bootstrap required, opened a new log
I20260926 09:32:43.438365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 3295
I20260926 09:32:43.438493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:43.438679 392 external_mini_cluster.cc:1428] Setting key d8333cc75a31822418c4981b6c8a1003
I20260926 09:32:43.439396 3363 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf28fdf4de4649dd8bb5c433c1067e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } }
I20260926 09:32:43.439589 3363 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:43.439634 3363 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf28fdf4de4649dd8bb5c433c1067e70, State: Initialized, Role: FOLLOWER
I20260926 09:32:43.439774 3363 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf28fdf4de4649dd8bb5c433c1067e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } }
I20260926 09:32:43.439864 3363 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:43.439917 3363 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:43.439976 3363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:43.440711 3363 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf28fdf4de4649dd8bb5c433c1067e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } }
I20260926 09:32:43.440805 3363 leader_election.cc:302] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf28fdf4de4649dd8bb5c433c1067e70; no voters:
I20260926 09:32:43.440994 3363 leader_election.cc:288] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:43.441051 3368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:43.441239 3368 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [term 1 LEADER]: Becoming Leader. State: Replica: cf28fdf4de4649dd8bb5c433c1067e70, State: Running, Role: LEADER
I20260926 09:32:43.441293 3363 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:43.441416 3368 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf28fdf4de4649dd8bb5c433c1067e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } }
I20260926 09:32:43.442718 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf28fdf4de4649dd8bb5c433c1067e70. Latest consensus state: current_term: 1 leader_uuid: "cf28fdf4de4649dd8bb5c433c1067e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf28fdf4de4649dd8bb5c433c1067e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } }
I20260926 09:32:43.442821 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:43.442725 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf28fdf4de4649dd8bb5c433c1067e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf28fdf4de4649dd8bb5c433c1067e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } }
I20260926 09:32:43.443100 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:43.443156 3378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:43.443715 3378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:43.445793 3378 catalog_manager.cc:1347] Generated new cluster ID: 275ba317126c43d0a80b8a60410025c7
I20260926 09:32:43.445850 3378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:32:43 dist-test-slave-21h8 krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:43.451850 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:43.439280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.0.98.62:41769)
0926 09:32:43.439947 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:43.439974 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:43.440012 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:32:43.440211 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:43.440219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:43.440277 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:43.440437 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:43.441266 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:43.442695 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:43.443992 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:43.444575 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:43.444666 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:43.447743 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:43.447748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:43.447748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:43.447762 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:43.449848 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:43.450345 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:43.450348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:43.450349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:43.450540 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:43.450785 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:43.450786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:43.450787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:43.451148 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:43.451354 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:43.451463 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 09:32:43.451621 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":560,"threads_started":1}
I20260926 09:32:43.452913 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:32:43.458181 3378 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:43.458945 3378 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:43.472772 3378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf28fdf4de4649dd8bb5c433c1067e70: Generated new TSK 0
I20260926 09:32:43.473109 3378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:32:43.474408 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:43.470452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.0.98.62:41769)
0926 09:32:43.470504 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:43.470506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:43.470642 (+ 136us) server_negotiation.cc:407] Connection header received
0926 09:32:43.470773 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:43.470774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:43.470788 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:43.470866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:43.471268 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:43.471873 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:43.472834 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:43.473070 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:43.473171 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:43.473912 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:43.473914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:43.473914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:43.474115 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:43.474131 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:43.474138 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:43.474211 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:43.474289 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 09:32:43.474316 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:32:43.477289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 3295
2026-09-26T09:32:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1345 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:43 dist-test-slave-21h8 krb5kdc[3396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:43 dist-test-slave-21h8 krb5kdc[3396](info): set up 2 sockets
Sep 26 09:32:43 dist-test-slave-21h8 krb5kdc[3396](info): commencing operation
krb5kdc: starting...
W20260926 09:32:44.523267 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:44 dist-test-slave-21h8 krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:44.651521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35295
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:44.735056 3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:44.735387 3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:44.735435 3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:44.736654 3409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:44.736753 3409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:44.736788 3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:44.736814 3409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:44.736832 3409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:44.738382 3409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:44.738711 3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:44.738965 3409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:44.744222 3415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:44.744232 3414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:44.744275 3417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:44.744580 3409 server_base.cc:1034] running on GCE node
I20260926 09:32:44.744786 3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:44.745023 3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:44.746162 3409 hybrid_clock.cc:648] HybridClock initialized: now 1790415164746150 us; error 28 us; skew 500 ppm
Sep 26 09:32:44 dist-test-slave-21h8 krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:44.748785 3409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:44.749226 3409 webserver.cc:460] Webserver started at http://127.0.98.62:35219/ using document root <none> and password file <none>
I20260926 09:32:44.749464 3409 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:44.749511 3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:44.749650 3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:44.750710 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "eff32b7e8f31440eb31f4b3449025b85"
format_stamp: "Formatted at 2026-09-26 09:32:44 on dist-test-slave-21h8"
server_key: "c1af4efcb044ea8e19427e340d5c5e33"
server_key_iv: "ce9debe7e71ef29458e0bd855e1b09f2"
server_key_version: "encryptionkey@0"
I20260926 09:32:44.751075 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "eff32b7e8f31440eb31f4b3449025b85"
format_stamp: "Formatted at 2026-09-26 09:32:44 on dist-test-slave-21h8"
server_key: "c1af4efcb044ea8e19427e340d5c5e33"
server_key_iv: "ce9debe7e71ef29458e0bd855e1b09f2"
server_key_version: "encryptionkey@0"
I20260926 09:32:44.752508 3409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:44.753257 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:44.753430 3409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:44.753535 3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "eff32b7e8f31440eb31f4b3449025b85"
format_stamp: "Formatted at 2026-09-26 09:32:44 on dist-test-slave-21h8"
server_key: "c1af4efcb044ea8e19427e340d5c5e33"
server_key_iv: "ce9debe7e71ef29458e0bd855e1b09f2"
server_key_version: "encryptionkey@0"
I20260926 09:32:44.753618 3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:44.768185 3409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:44.770828 3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:44.770953 3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:44.775180 3409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35295
I20260926 09:32:44.775220 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35295 every 8 connection(s)
I20260926 09:32:44.775650 3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:44.776543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 3409
I20260926 09:32:44.776646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:44.776858 392 external_mini_cluster.cc:1428] Setting key eb8564d69a6ec0a43368541e27767419
I20260926 09:32:44.776921 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:44.779963 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85: Bootstrap starting.
I20260926 09:32:44.780627 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:44.780923 3477 log.cc:824] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:44.781636 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85: No bootstrap required, opened a new log
I20260926 09:32:44.783576 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff32b7e8f31440eb31f4b3449025b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35295 } }
Sep 26 09:32:44 dist-test-slave-21h8 krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:44.783753 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:44.783802 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff32b7e8f31440eb31f4b3449025b85, State: Initialized, Role: FOLLOWER
I20260926 09:32:44.783957 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff32b7e8f31440eb31f4b3449025b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35295 } }
I20260926 09:32:44.784072 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:44.784118 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:44.784170 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:44.784986 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff32b7e8f31440eb31f4b3449025b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35295 } }
I20260926 09:32:44.785110 3477 leader_election.cc:302] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eff32b7e8f31440eb31f4b3449025b85; no voters:
I20260926 09:32:44.785300 3477 leader_election.cc:288] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:44.785506 3482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:44.785629 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:44.785732 3482 raft_consensus.cc:686] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [term 1 LEADER]: Becoming Leader. State: Replica: eff32b7e8f31440eb31f4b3449025b85, State: Running, Role: LEADER
I20260926 09:32:44.785862 3482 consensus_queue.cc:237] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff32b7e8f31440eb31f4b3449025b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35295 } }
I20260926 09:32:44.786276 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eff32b7e8f31440eb31f4b3449025b85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff32b7e8f31440eb31f4b3449025b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35295 } } }
I20260926 09:32:44.786353 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:44.786509 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eff32b7e8f31440eb31f4b3449025b85. Latest consensus state: current_term: 1 leader_uuid: "eff32b7e8f31440eb31f4b3449025b85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff32b7e8f31440eb31f4b3449025b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35295 } } }
I20260926 09:32:44.786562 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:44.786880 3491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:44.787524 3491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:44.788410 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:44.777477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.0.98.62:35295)
0926 09:32:44.777676 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:44.777691 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:44.777718 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:32:44.778372 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:44.778382 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:44.778438 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:44.778615 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:44.779263 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:44.780287 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:44.781176 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:44.781605 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:44.781744 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:44.784324 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:44.784329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:44.784329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:44.784344 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:44.786533 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:44.787040 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:44.787044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:44.787045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:44.787155 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:44.787438 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:44.787438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:44.787438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:44.787753 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:44.787942 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:44.788042 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 09:32:44.788185 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260926 09:32:44.789181 3491 catalog_manager.cc:1347] Generated new cluster ID: b16699827ac74706bd3d7eea8fa6d423
I20260926 09:32:44.789237 3491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:44.803499 3491 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:44.804307 3491 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:44.809461 3491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eff32b7e8f31440eb31f4b3449025b85: Generated new TSK 0
I20260926 09:32:44.809711 3491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:32:44.843593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35295
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:44.937667 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:44.937892 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:44.937919 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:44.939292 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:44.939437 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:44.939504 3503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:44.941155 3503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:44.941439 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:44.941711 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:44.947144 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:44.947147 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:44.947335 3503 server_base.cc:1034] running on GCE node
W20260926 09:32:44.947216 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:44.947679 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:44.947975 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:44.949170 3503 hybrid_clock.cc:648] HybridClock initialized: now 1790415164949139 us; error 44 us; skew 500 ppm
Sep 26 09:32:44 dist-test-slave-21h8 krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:44.951982 3503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:44.952409 3503 webserver.cc:460] Webserver started at http://127.0.98.1:35155/ using document root <none> and password file <none>
I20260926 09:32:44.952652 3503 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:44.952699 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:44.952862 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:44.953953 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0a197375ad94b8ea466159a0ecb1aad"
format_stamp: "Formatted at 2026-09-26 09:32:44 on dist-test-slave-21h8"
server_key: "32c90ac298929ca0d89d92ea034089a0"
server_key_iv: "e884108f94f8189c55f895a5dd0ae488"
server_key_version: "encryptionkey@0"
I20260926 09:32:44.954342 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0a197375ad94b8ea466159a0ecb1aad"
format_stamp: "Formatted at 2026-09-26 09:32:44 on dist-test-slave-21h8"
server_key: "32c90ac298929ca0d89d92ea034089a0"
server_key_iv: "e884108f94f8189c55f895a5dd0ae488"
server_key_version: "encryptionkey@0"
I20260926 09:32:44.955866 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:32:44.956772 3518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:44.956971 3503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:44.957079 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "d0a197375ad94b8ea466159a0ecb1aad"
format_stamp: "Formatted at 2026-09-26 09:32:44 on dist-test-slave-21h8"
server_key: "32c90ac298929ca0d89d92ea034089a0"
server_key_iv: "e884108f94f8189c55f895a5dd0ae488"
server_key_version: "encryptionkey@0"
I20260926 09:32:44.957190 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:44.971223 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:44.973919 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:44.974104 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:44.974375 3503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:44.974712 3503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:44.974773 3503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:44.974815 3503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:44.974835 3503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:44.980856 3503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45811
I20260926 09:32:44.980899 3631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45811 every 8 connection(s)
I20260926 09:32:44.981278 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:32:44 dist-test-slave-21h8 krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:44.990119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 3503
I20260926 09:32:44.990257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:44.990483 392 external_mini_cluster.cc:1428] Setting key 18e320e8b2b8b68af2b7b8c0296aa38a
I20260926 09:32:44.991570 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:44.982068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52115 (local address 127.0.98.62:35295)
0926 09:32:44.982119 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:44.982120 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:44.982349 (+ 229us) server_negotiation.cc:407] Connection header received
0926 09:32:44.982828 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:44.982829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:44.982839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:44.982897 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:44.983974 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:44.984513 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:44.985595 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:44.985876 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:44.985937 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:44.988524 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:44.988526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:44.988526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:44.988534 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:44.989849 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:44.990330 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:44.990332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:44.990332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:44.990403 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:44.990738 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:44.990740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:44.990740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:44.990900 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:44.991013 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:44.991402 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 09:32:44.991506 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:32:44.991930 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:44.982057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35295 (local address 127.0.98.1:52115)
0926 09:32:44.982216 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:44.982254 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:44.982739 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:44.982953 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:44.982956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:44.983650 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:44.983860 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:44.983865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:44.984618 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:44.984619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:44.985508 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:44.985511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:44.985966 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:44.985966 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:44.986080 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:44.986824 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:44.986834 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:44.988320 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:44.989935 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:44.989942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:44.989944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:44.990184 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:44.990482 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:44.990483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:44.990484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:44.990648 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:44.991004 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:44.991011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:44.991220 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 09:32:44.991371 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 09:32:44.991595 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260926 09:32:44.992450 3632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35295
I20260926 09:32:44.992563 3632 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:44.992774 3632 heartbeater.cc:511] Master 127.0.98.62:35295 requested a full tablet report, sending...
I20260926 09:32:44.993335 3441 ts_manager.cc:194] Registered new tserver with Master: d0a197375ad94b8ea466159a0ecb1aad (127.0.98.1:45811)
I20260926 09:32:44.993985 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52115
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:45.019546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35295
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:45.105413 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:45.105626 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:45.105654 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:45.106884 3637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:45.106930 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:45.106959 3637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:45.108480 3637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:45.108707 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:45.108902 3637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:45.114224 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:45.114235 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:45.114234 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:45.114509 3637 server_base.cc:1034] running on GCE node
I20260926 09:32:45.114743 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:45.115023 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:45.116256 3637 hybrid_clock.cc:648] HybridClock initialized: now 1790415165116256 us; error 49 us; skew 500 ppm
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:45.119073 3637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:45.119603 3637 webserver.cc:460] Webserver started at http://127.0.98.2:46625/ using document root <none> and password file <none>
I20260926 09:32:45.119879 3637 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:45.119928 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:45.120086 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:45.121208 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "df49d02cad814a5293801504668763e3"
format_stamp: "Formatted at 2026-09-26 09:32:45 on dist-test-slave-21h8"
server_key: "ad9bd99dd78a8e7e6a8fa93fb3760d67"
server_key_iv: "960ac3bc80d3daced4ae36028d8714de"
server_key_version: "encryptionkey@0"
I20260926 09:32:45.121570 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "df49d02cad814a5293801504668763e3"
format_stamp: "Formatted at 2026-09-26 09:32:45 on dist-test-slave-21h8"
server_key: "ad9bd99dd78a8e7e6a8fa93fb3760d67"
server_key_iv: "960ac3bc80d3daced4ae36028d8714de"
server_key_version: "encryptionkey@0"
I20260926 09:32:45.123009 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:45.123950 3652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:45.124236 3637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:45.124320 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "df49d02cad814a5293801504668763e3"
format_stamp: "Formatted at 2026-09-26 09:32:45 on dist-test-slave-21h8"
server_key: "ad9bd99dd78a8e7e6a8fa93fb3760d67"
server_key_iv: "960ac3bc80d3daced4ae36028d8714de"
server_key_version: "encryptionkey@0"
I20260926 09:32:45.124377 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:45.143760 3637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:45.145905 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:45.146087 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:45.146299 3637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:45.146669 3637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:45.146705 3637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:45.146728 3637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:45.146786 3637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:45.152626 3637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42781
I20260926 09:32:45.152690 3765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42781 every 8 connection(s)
I20260926 09:32:45.153019 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:45.154944 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 3637
I20260926 09:32:45.155082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:45.155300 392 external_mini_cluster.cc:1428] Setting key 87b1f3b7fda0a45440a58315995c274d
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:32:45.164248 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:45.153777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51667 (local address 127.0.98.62:35295)
0926 09:32:45.153821 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:45.153824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:45.154050 (+ 226us) server_negotiation.cc:407] Connection header received
0926 09:32:45.154593 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:45.154594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:45.154604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:45.154658 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:45.156002 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.156812 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.158213 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.158400 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.158466 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:45.161015 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:45.161016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:45.161016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:45.161025 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:45.162659 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:45.163126 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:45.163127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:45.163128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:45.163194 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:45.163516 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:45.163516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:45.163517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:45.163658 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:45.163736 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:45.164057 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 09:32:45.164151 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:32:45.164470 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:45.153749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35295 (local address 127.0.98.2:51667)
0926 09:32:45.153903 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:45.153941 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:45.154491 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:45.154748 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:45.154753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:45.155609 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:45.155916 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:45.155923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.156933 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.156934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:45.158100 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:45.158103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.158495 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.158496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:45.158622 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:45.159188 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:45.159197 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:45.160894 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:45.162771 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:45.162778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:45.162780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:45.163021 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:45.163265 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:45.163266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:45.163266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:45.163431 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:45.163739 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:45.163746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:45.163905 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:32:45.164016 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 09:32:45.164212 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 09:32:45.165036 3766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35295
I20260926 09:32:45.165171 3766 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:45.165364 3766 heartbeater.cc:511] Master 127.0.98.62:35295 requested a full tablet report, sending...
I20260926 09:32:45.165802 3441 ts_manager.cc:194] Registered new tserver with Master: df49d02cad814a5293801504668763e3 (127.0.98.2:42781)
I20260926 09:32:45.166226 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:45.186385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35295
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:45.269261 3771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:45.269493 3771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:45.269519 3771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:45.270684 3771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:45.270726 3771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:45.270753 3771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:45.272256 3771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:45.272462 3771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:45.272644 3771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:45.278273 3777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:45.278314 3776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:45.278334 3771 server_base.cc:1034] running on GCE node
W20260926 09:32:45.278292 3779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:45.278738 3771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:45.279052 3771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:45.280241 3771 hybrid_clock.cc:648] HybridClock initialized: now 1790415165280228 us; error 44 us; skew 500 ppm
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:45.283023 3771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:45.283638 3771 webserver.cc:460] Webserver started at http://127.0.98.3:40979/ using document root <none> and password file <none>
I20260926 09:32:45.283888 3771 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:45.283962 3771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:45.284112 3771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:45.285308 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bfde89bade24c5a843712b1238dea67"
format_stamp: "Formatted at 2026-09-26 09:32:45 on dist-test-slave-21h8"
server_key: "5cd92e62047e1072e435648fd381abca"
server_key_iv: "d4aa28f91cd192717b02a9bfdfc4bddb"
server_key_version: "encryptionkey@0"
I20260926 09:32:45.285709 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bfde89bade24c5a843712b1238dea67"
format_stamp: "Formatted at 2026-09-26 09:32:45 on dist-test-slave-21h8"
server_key: "5cd92e62047e1072e435648fd381abca"
server_key_iv: "d4aa28f91cd192717b02a9bfdfc4bddb"
server_key_version: "encryptionkey@0"
I20260926 09:32:45.287366 3771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 09:32:45.288323 3786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:45.288632 3771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:45.288721 3771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "8bfde89bade24c5a843712b1238dea67"
format_stamp: "Formatted at 2026-09-26 09:32:45 on dist-test-slave-21h8"
server_key: "5cd92e62047e1072e435648fd381abca"
server_key_iv: "d4aa28f91cd192717b02a9bfdfc4bddb"
server_key_version: "encryptionkey@0"
I20260926 09:32:45.288780 3771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:45.298514 3771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:45.301317 3771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:45.301438 3771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:45.301702 3771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:45.302009 3771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:45.302049 3771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:45.302076 3771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:45.302089 3771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:45.308733 3771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43485
I20260926 09:32:45.308785 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43485 every 8 connection(s)
I20260926 09:32:45.309254 3771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:32:45.311875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 3771
I20260926 09:32:45.311986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:45.312186 392 external_mini_cluster.cc:1428] Setting key 76f304482e543a58ce1f4ea5f9ab81e0
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:45.319221 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:45.310436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51303 (local address 127.0.98.62:35295)
0926 09:32:45.310506 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:45.310508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:45.310621 (+ 113us) server_negotiation.cc:407] Connection header received
0926 09:32:45.311167 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:45.311168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:45.311179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:45.311229 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:45.312411 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.313080 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.314121 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.314317 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.314373 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:45.316622 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:45.316623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:45.316623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:45.316630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:45.317880 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:45.318316 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:45.318317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:45.318317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:45.318383 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:45.318620 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:45.318621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:45.318621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:45.318738 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:45.318792 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:45.319051 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 09:32:45.319134 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:32:45.319552 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:45.310201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35295 (local address 127.0.98.3:51303)
0926 09:32:45.310443 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:45.310502 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:45.311064 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:45.311294 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:45.311298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:45.312061 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:45.312327 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:45.312333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.313182 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.313183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:45.314038 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:45.314041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.314393 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.314394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:45.314493 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:45.314990 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:45.315000 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:45.316528 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:45.317978 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:45.317984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:45.317986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:45.318220 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:45.318455 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:45.318456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:45.318456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:45.318557 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:45.318799 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:45.318803 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:45.318921 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 09:32:45.319015 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 09:32:45.319174 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260926 09:32:45.320148 3900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35295
I20260926 09:32:45.320276 3900 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:45.320461 3900 heartbeater.cc:511] Master 127.0.98.62:35295 requested a full tablet report, sending...
I20260926 09:32:45.320885 3441 ts_manager.cc:194] Registered new tserver with Master: 8bfde89bade24c5a843712b1238dea67 (127.0.98.3:43485)
I20260926 09:32:45.321301 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51303
I20260926 09:32:45.324096 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415165, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415165, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:45.353842 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:45.340436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.0.98.62:35295)
0926 09:32:45.340491 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:45.340494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:45.343497 (+ 3003us) server_negotiation.cc:407] Connection header received
0926 09:32:45.343625 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:45.343626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:45.343640 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:45.343682 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:45.344582 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.345389 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.346420 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:45.346694 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:45.346743 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:45.349989 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:45.349991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:45.349991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:45.350003 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:45.352005 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:45.352474 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:45.352475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:45.352476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:45.352566 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:45.352789 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:45.352789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:45.352790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:45.353035 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:45.353144 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:45.353576 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 09:32:45.353695 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 09:32:45.354089 3441 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52790
I20260926 09:32:45.354750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 3503
I20260926 09:32:45.359935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 3637
I20260926 09:32:45.364354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 3771
I20260926 09:32:45.368799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 3409
2026-09-26T09:32:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1893 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3912](info): set up 2 sockets
Sep 26 09:32:45 dist-test-slave-21h8 krb5kdc[3912](info): commencing operation
krb5kdc: starting...
W20260926 09:32:46.403278 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:46 dist-test-slave-21h8 krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:46.531970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:46.615561 3925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:46.615846 3925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:46.615890 3925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:46.617129 3925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:46.617214 3925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:46.617240 3925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:46.617264 3925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:46.617290 3925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:46.618824 3925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:46.619131 3925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:46.619415 3925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:46.624598 3931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:46.624679 3933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:46.624601 3930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:46.624779 3925 server_base.cc:1034] running on GCE node
I20260926 09:32:46.625119 3925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:46.625481 3925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:46.626653 3925 hybrid_clock.cc:648] HybridClock initialized: now 1790415166626639 us; error 52 us; skew 500 ppm
Sep 26 09:32:46 dist-test-slave-21h8 krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:46.629436 3925 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:46.629925 3925 webserver.cc:460] Webserver started at http://127.0.98.62:35903/ using document root <none> and password file <none>
I20260926 09:32:46.630188 3925 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:46.630263 3925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:46.630414 3925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:46.631595 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "081c73ddcfb0482fbae879e0ccfb6e8e"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "bd4285628e9919c0ec88f268eee0ca1c"
server_key_iv: "f204e761a2760b2212e43ddbb7034915"
server_key_version: "encryptionkey@0"
I20260926 09:32:46.631975 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "081c73ddcfb0482fbae879e0ccfb6e8e"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "bd4285628e9919c0ec88f268eee0ca1c"
server_key_iv: "f204e761a2760b2212e43ddbb7034915"
server_key_version: "encryptionkey@0"
I20260926 09:32:46.633428 3925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:46.634253 3940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:46.634449 3925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:46.634562 3925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "081c73ddcfb0482fbae879e0ccfb6e8e"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "bd4285628e9919c0ec88f268eee0ca1c"
server_key_iv: "f204e761a2760b2212e43ddbb7034915"
server_key_version: "encryptionkey@0"
I20260926 09:32:46.634657 3925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:46.654543 3925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:46.657313 3925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:46.657495 3925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:46.661919 3925 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34353
I20260926 09:32:46.661978 3992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34353 every 8 connection(s)
I20260926 09:32:46.662382 3925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:46.663296 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:46.665711 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e: Bootstrap starting.
I20260926 09:32:46.666262 3993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:46.666508 3993 log.cc:824] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:46.667246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 3925
I20260926 09:32:46.667354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:46.667388 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e: No bootstrap required, opened a new log
I20260926 09:32:46.667593 392 external_mini_cluster.cc:1428] Setting key 9768af48a4b333eac6a2d842c4cae036
I20260926 09:32:46.669984 3993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c73ddcfb0482fbae879e0ccfb6e8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34353 } }
I20260926 09:32:46.670167 3993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:46.670218 3993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081c73ddcfb0482fbae879e0ccfb6e8e, State: Initialized, Role: FOLLOWER
I20260926 09:32:46.670354 3993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c73ddcfb0482fbae879e0ccfb6e8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34353 } }
I20260926 09:32:46.670451 3993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:46.670495 3993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:46.670543 3993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:46.671126 3993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c73ddcfb0482fbae879e0ccfb6e8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34353 } }
I20260926 09:32:46.671255 3993 leader_election.cc:302] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 081c73ddcfb0482fbae879e0ccfb6e8e; no voters:
I20260926 09:32:46.671487 3993 leader_election.cc:288] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:46.671566 3998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:46.671782 3993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:46.671780 3998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [term 1 LEADER]: Becoming Leader. State: Replica: 081c73ddcfb0482fbae879e0ccfb6e8e, State: Running, Role: LEADER
I20260926 09:32:46.671998 3998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c73ddcfb0482fbae879e0ccfb6e8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34353 } }
I20260926 09:32:46.672422 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "081c73ddcfb0482fbae879e0ccfb6e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c73ddcfb0482fbae879e0ccfb6e8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34353 } } }
I20260926 09:32:46.672415 3998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 081c73ddcfb0482fbae879e0ccfb6e8e. Latest consensus state: current_term: 1 leader_uuid: "081c73ddcfb0482fbae879e0ccfb6e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081c73ddcfb0482fbae879e0ccfb6e8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34353 } } }
I20260926 09:32:46.672595 3998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:46.672597 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:46.673573 4007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:46.674135 4007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:46.675736 4007 catalog_manager.cc:1347] Generated new cluster ID: ca5fa01c36414126ab8168ab8024f0ae
Sep 26 09:32:46 dist-test-slave-21h8 krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:46.675793 4007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:46.680352 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:46.668184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.0.98.62:34353)
0926 09:32:46.668385 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:46.668393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:46.668419 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:32:46.668862 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:46.668870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:46.668920 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:46.669044 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:46.669868 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:46.671214 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:46.672392 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:46.673107 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:46.673201 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:46.676366 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:46.676370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:46.676370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:46.676383 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:46.678521 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:46.678952 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:46.678955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:46.678956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:46.679074 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:46.679306 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:46.679307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:46.679307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:46.679678 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:46.679867 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:46.679945 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 09:32:46.680110 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260926 09:32:46.692823 4007 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:46.693651 4007 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:46.698048 4007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 081c73ddcfb0482fbae879e0ccfb6e8e: Generated new TSK 0
I20260926 09:32:46.698269 4007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:32:46.727680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34353
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:46.814368 4019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:46.814599 4019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:46.814625 4019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:46.815899 4019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:46.815945 4019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:46.815981 4019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:46.817520 4019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:46.817785 4019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:46.818058 4019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:46.823647 4025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:46.823668 4027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:46.823771 4019 server_base.cc:1034] running on GCE node
W20260926 09:32:46.823660 4024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:46.824054 4019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:46.824301 4019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:46.825453 4019 hybrid_clock.cc:648] HybridClock initialized: now 1790415166825457 us; error 49 us; skew 500 ppm
Sep 26 09:32:46 dist-test-slave-21h8 krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:46.828286 4019 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:46.828776 4019 webserver.cc:460] Webserver started at http://127.0.98.1:37333/ using document root <none> and password file <none>
I20260926 09:32:46.829028 4019 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:46.829103 4019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:46.829277 4019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:46.830420 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b6b1e2046b846c1a49fb3cf72a7fb5d"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "ba1de3dcb24c45b0f95b2406c2b071f1"
server_key_iv: "1f583e287a525f85992438d1d45d7a8f"
server_key_version: "encryptionkey@0"
I20260926 09:32:46.830808 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b6b1e2046b846c1a49fb3cf72a7fb5d"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "ba1de3dcb24c45b0f95b2406c2b071f1"
server_key_iv: "1f583e287a525f85992438d1d45d7a8f"
server_key_version: "encryptionkey@0"
I20260926 09:32:46.832319 4019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:32:46.833144 4034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:46.833324 4019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:46.833431 4019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "0b6b1e2046b846c1a49fb3cf72a7fb5d"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "ba1de3dcb24c45b0f95b2406c2b071f1"
server_key_iv: "1f583e287a525f85992438d1d45d7a8f"
server_key_version: "encryptionkey@0"
I20260926 09:32:46.833515 4019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:46.845072 4019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:46.847258 4019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:46.847468 4019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:46.847724 4019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:46.848042 4019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:46.848104 4019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:46.848151 4019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:46.848174 4019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:46.854110 4019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38039
I20260926 09:32:46.854154 4147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38039 every 8 connection(s)
I20260926 09:32:46.854489 4019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:32:46 dist-test-slave-21h8 krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:46.863070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 4019
I20260926 09:32:46.863189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:46.863415 392 external_mini_cluster.cc:1428] Setting key 9037c9f698666f9ad3710e2ce89a5bdb
I20260926 09:32:46.864691 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:46.855181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34353 (local address 127.0.98.1:37183)
0926 09:32:46.855501 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:46.855538 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:46.856503 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:46.856808 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:46.856812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:46.857085 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:46.857282 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:46.857286 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:46.858025 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:46.858026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:46.858860 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:46.858862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:46.859294 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:46.859295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:46.859453 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:46.859984 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:46.859993 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:46.861503 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:46.863040 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:46.863048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:46.863051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:46.863420 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:46.863653 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:46.863654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:46.863654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:46.863762 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:46.864116 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:46.864124 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:46.864289 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 09:32:46.864377 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:32:46.864508 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":230,"threads_started":1}
I20260926 09:32:46.864691 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:46.855225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37183 (local address 127.0.98.62:34353)
0926 09:32:46.855383 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:46.855386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:46.855629 (+ 243us) server_negotiation.cc:407] Connection header received
0926 09:32:46.856644 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:46.856646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:46.856658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:46.856716 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:46.857392 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:46.857917 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:46.858965 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:46.859187 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:46.859281 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:46.861597 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:46.861599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:46.861599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:46.861606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:46.862940 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:46.863507 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:46.863508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:46.863508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:46.863571 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:46.863875 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:46.863876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:46.863876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:46.863991 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:46.864081 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:46.864450 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 09:32:46.864551 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 09:32:46.865247 4148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34353
I20260926 09:32:46.865360 4148 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:46.865569 4148 heartbeater.cc:511] Master 127.0.98.62:34353 requested a full tablet report, sending...
I20260926 09:32:46.866292 3957 ts_manager.cc:194] Registered new tserver with Master: 0b6b1e2046b846c1a49fb3cf72a7fb5d (127.0.98.1:38039)
I20260926 09:32:46.867118 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37183
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:46.894302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34353
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:46.980734 4153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:46.981006 4153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:46.981050 4153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:46.982311 4153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:46.982393 4153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:46.982442 4153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:46.984190 4153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:46.984493 4153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:46.984757 4153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:46.990506 4158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:46.990660 4153 server_base.cc:1034] running on GCE node
W20260926 09:32:46.990461 4161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:46.990473 4159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:46.991021 4153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:46.991389 4153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:46.992558 4153 hybrid_clock.cc:648] HybridClock initialized: now 1790415166992547 us; error 28 us; skew 500 ppm
Sep 26 09:32:46 dist-test-slave-21h8 krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:46.995476 4153 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:46.995919 4153 webserver.cc:460] Webserver started at http://127.0.98.2:46279/ using document root <none> and password file <none>
I20260926 09:32:46.996188 4153 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:46.996266 4153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:46.996423 4153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:46.997509 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "ecdab849768449028a2c40ef15c4b268"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "389623c2318c24b3eac72b4bd18778c0"
server_key_iv: "c6f1a3fa42aec0ced023a743ac84223b"
server_key_version: "encryptionkey@0"
I20260926 09:32:46.997910 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ecdab849768449028a2c40ef15c4b268"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "389623c2318c24b3eac72b4bd18778c0"
server_key_iv: "c6f1a3fa42aec0ced023a743ac84223b"
server_key_version: "encryptionkey@0"
I20260926 09:32:46.999405 4153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:47.000358 4168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:47.000643 4153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:47.000782 4153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "ecdab849768449028a2c40ef15c4b268"
format_stamp: "Formatted at 2026-09-26 09:32:46 on dist-test-slave-21h8"
server_key: "389623c2318c24b3eac72b4bd18778c0"
server_key_iv: "c6f1a3fa42aec0ced023a743ac84223b"
server_key_version: "encryptionkey@0"
I20260926 09:32:47.000875 4153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:47.034902 4153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:47.037773 4153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:47.037954 4153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:47.038352 4153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:47.038717 4153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:47.038760 4153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:47.038792 4153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:47.038810 4153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:47.044872 4153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43949
I20260926 09:32:47.044937 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43949 every 8 connection(s)
I20260926 09:32:47.045286 4153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:47.050032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 4153
I20260926 09:32:47.050128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:47.050320 392 external_mini_cluster.cc:1428] Setting key 12bc09e81ba60e99c0ed0161fbad52ea
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:47.056031 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:47.046215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33053 (local address 127.0.98.62:34353)
0926 09:32:47.046283 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:47.046285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:47.046556 (+ 271us) server_negotiation.cc:407] Connection header received
0926 09:32:47.047595 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:47.047597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:47.047610 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:47.047650 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:47.048324 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.048869 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.049961 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.050234 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.050298 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:47.052942 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:47.052944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:47.052944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:47.052953 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:47.054458 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:47.054918 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:47.054919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:47.054920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:47.054987 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:47.055271 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:47.055272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:47.055272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:47.055463 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:47.055549 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:47.055843 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 09:32:47.055934 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:32:47.056274 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:47.046204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34353 (local address 127.0.98.2:33053)
0926 09:32:47.046405 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:47.046448 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:47.047455 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:47.047736 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:47.047740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:47.048011 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:47.048221 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:47.048225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.048986 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.048987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:47.049856 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:47.049859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.050332 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.050334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:47.050499 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:47.051108 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:47.051119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:47.052828 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:47.054572 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:47.054579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:47.054581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:47.054832 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:47.055071 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:47.055072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:47.055072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:47.055191 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:47.055539 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:47.055546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:47.055700 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 09:32:47.055820 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 09:32:47.055990 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260926 09:32:47.056847 4282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34353
I20260926 09:32:47.056950 4282 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:47.057178 4282 heartbeater.cc:511] Master 127.0.98.62:34353 requested a full tablet report, sending...
I20260926 09:32:47.057621 3957 ts_manager.cc:194] Registered new tserver with Master: ecdab849768449028a2c40ef15c4b268 (127.0.98.2:43949)
I20260926 09:32:47.058176 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33053
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:47.078822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34353
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:47.162468 4287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:47.162703 4287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:47.162730 4287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:47.164016 4287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:47.164063 4287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:47.164093 4287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:47.165670 4287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:47.165895 4287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:47.166103 4287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:47.171916 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:47.171995 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:47.172019 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:47.172158 4287 server_base.cc:1034] running on GCE node
I20260926 09:32:47.172399 4287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:47.172684 4287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:47.173894 4287 hybrid_clock.cc:648] HybridClock initialized: now 1790415167173883 us; error 28 us; skew 500 ppm
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:47.176805 4287 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:47.177333 4287 webserver.cc:460] Webserver started at http://127.0.98.3:44487/ using document root <none> and password file <none>
I20260926 09:32:47.177608 4287 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:47.177683 4287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:47.177843 4287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:47.178942 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "314ec481e78f45e8989e1284a94a21a3"
format_stamp: "Formatted at 2026-09-26 09:32:47 on dist-test-slave-21h8"
server_key: "d7512e977163b9fd1183608ce3a2a5dd"
server_key_iv: "c909ff7debe45216dd3ad17f90469679"
server_key_version: "encryptionkey@0"
I20260926 09:32:47.179384 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "314ec481e78f45e8989e1284a94a21a3"
format_stamp: "Formatted at 2026-09-26 09:32:47 on dist-test-slave-21h8"
server_key: "d7512e977163b9fd1183608ce3a2a5dd"
server_key_iv: "c909ff7debe45216dd3ad17f90469679"
server_key_version: "encryptionkey@0"
I20260926 09:32:47.180999 4287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:47.181838 4302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:47.182044 4287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:47.182154 4287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "314ec481e78f45e8989e1284a94a21a3"
format_stamp: "Formatted at 2026-09-26 09:32:47 on dist-test-slave-21h8"
server_key: "d7512e977163b9fd1183608ce3a2a5dd"
server_key_iv: "c909ff7debe45216dd3ad17f90469679"
server_key_version: "encryptionkey@0"
I20260926 09:32:47.182240 4287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:47.193244 4287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:47.195807 4287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:47.195986 4287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:47.196239 4287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:47.196566 4287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:47.196625 4287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:47.196668 4287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:47.196689 4287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:47.203183 4287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38705
I20260926 09:32:47.203244 4415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38705 every 8 connection(s)
I20260926 09:32:47.203624 4287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:32:47.204062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 4287
I20260926 09:32:47.204162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:47.204355 392 external_mini_cluster.cc:1428] Setting key fd7b04bd5b4993d73ba94aa6c9888ff7
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:47.213426 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:47.204396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47435 (local address 127.0.98.62:34353)
0926 09:32:47.204447 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:47.204449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:47.204712 (+ 263us) server_negotiation.cc:407] Connection header received
0926 09:32:47.205898 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:47.205900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:47.205910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:47.205970 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:47.206618 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.207160 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.208256 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.208445 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.208514 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:47.210791 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:47.210793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:47.210793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:47.210800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:47.212215 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:47.212627 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:47.212628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:47.212628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:47.212677 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:47.212898 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:47.212898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:47.212898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:47.213003 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:47.213051 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:47.213295 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 09:32:47.213357 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:32:47.213747 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:47.204390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34353 (local address 127.0.98.3:47435)
0926 09:32:47.204556 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:47.204607 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:47.205775 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:47.206037 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:47.206042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:47.206311 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:47.206512 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:47.206516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.207254 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.207255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:47.208160 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:47.208162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.208533 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.208534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:47.208633 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:47.209117 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:47.209126 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:47.210701 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:47.212312 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:47.212319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:47.212321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:47.212552 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:47.212740 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:47.212741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:47.212742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:47.212842 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:47.213060 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:47.213065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:47.213182 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 09:32:47.213267 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:32:47.213388 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260926 09:32:47.214397 4416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34353
I20260926 09:32:47.214498 4416 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:47.214711 4416 heartbeater.cc:511] Master 127.0.98.62:34353 requested a full tablet report, sending...
I20260926 09:32:47.215194 3957 ts_manager.cc:194] Registered new tserver with Master: 314ec481e78f45e8989e1284a94a21a3 (127.0.98.3:38705)
I20260926 09:32:47.215724 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47435
I20260926 09:32:47.216270 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:47.239426 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:47.231528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.0.98.1:38039)
0926 09:32:47.231737 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:47.231742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:47.231755 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:32:47.231783 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:47.231785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:47.231799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:47.231869 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:47.232436 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.233173 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.233868 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.234205 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.234258 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:47.236646 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:47.236654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:47.236655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:47.236661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:47.238251 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:47.238585 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:47.238587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:47.238587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:47.238657 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:47.238880 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:47.238881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:47.238881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:47.239012 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:47.239124 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:47.239199 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:32:47.239264 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:47.247596 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:47.240742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.0.98.62:34353)
0926 09:32:47.240809 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:47.240811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:47.240821 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:47.240883 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:47.240884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:47.240892 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:47.240941 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:47.241380 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.241865 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.242531 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:47.242724 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:47.242796 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:47.245041 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:47.245043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:47.245044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:47.245050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:47.246549 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:47.246855 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:47.246856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:47.246856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:47.246903 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:47.247079 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:47.247080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:47.247080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:47.247218 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:47.247296 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:47.247465 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:32:47.247528 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 09:32:47.248008 3957 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766
I20260926 09:32:47.248466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 4019
I20260926 09:32:47.253073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 4153
I20260926 09:32:47.257817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 4287
I20260926 09:32:47.262095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 3925
2026-09-26T09:32:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1893 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[4428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[4428](info): set up 2 sockets
Sep 26 09:32:47 dist-test-slave-21h8 krb5kdc[4428](info): commencing operation
krb5kdc: starting...
W20260926 09:32:48.303188 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:48 dist-test-slave-21h8 krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:48.422429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37431
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:48.503553 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:48.503767 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:48.503789 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:48.504984 4441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:48.505029 4441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:48.505043 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:48.505055 4441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:48.505070 4441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:48.506589 4441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:48.506856 4441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:48.507097 4441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:48.512157 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:48.512163 4441 server_base.cc:1034] running on GCE node
W20260926 09:32:48.512143 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:48.512143 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:48.512672 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:48.512982 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:48.514132 4441 hybrid_clock.cc:648] HybridClock initialized: now 1790415168514123 us; error 31 us; skew 500 ppm
Sep 26 09:32:48 dist-test-slave-21h8 krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:48.522984 4441 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:48.523456 4441 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:48.523533 4441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:48.523689 4441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:48.524809 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "5ea105210bb24c8caf61ae68b1710dfc"
format_stamp: "Formatted at 2026-09-26 09:32:48 on dist-test-slave-21h8"
server_key: "69f3d1cdac750a0f8c92424f2ad6c35c"
server_key_iv: "842105c79bc84cf519e3868164bb80cc"
server_key_version: "encryptionkey@0"
I20260926 09:32:48.525192 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ea105210bb24c8caf61ae68b1710dfc"
format_stamp: "Formatted at 2026-09-26 09:32:48 on dist-test-slave-21h8"
server_key: "69f3d1cdac750a0f8c92424f2ad6c35c"
server_key_iv: "842105c79bc84cf519e3868164bb80cc"
server_key_version: "encryptionkey@0"
I20260926 09:32:48.526517 4441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:48.527258 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:48.527454 4441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:48.527559 4441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "5ea105210bb24c8caf61ae68b1710dfc"
format_stamp: "Formatted at 2026-09-26 09:32:48 on dist-test-slave-21h8"
server_key: "69f3d1cdac750a0f8c92424f2ad6c35c"
server_key_iv: "842105c79bc84cf519e3868164bb80cc"
server_key_version: "encryptionkey@0"
I20260926 09:32:48.527654 4441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:48.534451 4441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:48.536325 4441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:48.536494 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:48.540575 4441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37431
I20260926 09:32:48.540629 4507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37431 every 8 connection(s)
I20260926 09:32:48.540905 4441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:48.542029 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:48.545822 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc: Bootstrap starting.
I20260926 09:32:48.546550 4508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:48.546876 4508 log.cc:824] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:48.547192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 4441
I20260926 09:32:48.547307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:48.547555 392 external_mini_cluster.cc:1428] Setting key 43d9fbe7865f2025a6b8686500fce976
I20260926 09:32:48.547756 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc: No bootstrap required, opened a new log
I20260926 09:32:48.549697 4508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea105210bb24c8caf61ae68b1710dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260926 09:32:48.549876 4508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:48.549916 4508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea105210bb24c8caf61ae68b1710dfc, State: Initialized, Role: FOLLOWER
I20260926 09:32:48.550081 4508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea105210bb24c8caf61ae68b1710dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260926 09:32:48.550176 4508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:48.550210 4508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:48.550259 4508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 0 FOLLOWER]: Advancing to term 1
Sep 26 09:32:48 dist-test-slave-21h8 krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:48.558611 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:48.548165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.0.98.62:37431)
0926 09:32:48.548365 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:48.548374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:48.548403 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:32:48.549678 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:48.549685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:48.549736 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:48.549853 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:48.550685 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:48.551686 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:48.552406 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:48.552794 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:48.552878 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:48.555121 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:48.555124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:48.555125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:48.555136 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:48.556728 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:48.557063 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:48.557065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:48.557066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:48.557167 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:48.557345 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:48.557346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:48.557346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:48.557593 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:48.557690 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:48.558253 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 09:32:48.558414 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260926 09:32:48.584481 4508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea105210bb24c8caf61ae68b1710dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260926 09:32:48.584695 4508 leader_election.cc:302] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ea105210bb24c8caf61ae68b1710dfc; no voters:
I20260926 09:32:48.584919 4508 leader_election.cc:288] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:48.585060 4513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:48.585263 4513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [term 1 LEADER]: Becoming Leader. State: Replica: 5ea105210bb24c8caf61ae68b1710dfc, State: Running, Role: LEADER
I20260926 09:32:48.585271 4508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:48.585456 4513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea105210bb24c8caf61ae68b1710dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260926 09:32:48.586004 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ea105210bb24c8caf61ae68b1710dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea105210bb24c8caf61ae68b1710dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } }
I20260926 09:32:48.586297 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:48.586210 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ea105210bb24c8caf61ae68b1710dfc. Latest consensus state: current_term: 1 leader_uuid: "5ea105210bb24c8caf61ae68b1710dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea105210bb24c8caf61ae68b1710dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } }
I20260926 09:32:48.586593 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc [sys.catalog]: This master's current role is: LEADER
W20260926 09:32:48.587673 4528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:32:48.587733 4528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:32:48.587790 4523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:48.587982 4523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:48.589545 4523 catalog_manager.cc:1347] Generated new cluster ID: 0860db27ad46466c8708134025934567
I20260926 09:32:48.589630 4523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:48.598634 4523 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:48.599187 4523 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:48.607048 4523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc: Generated new TSK 0
I20260926 09:32:48.607277 4523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260926 09:32:49.078394 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.469s user 0.003s sys 0.000s
I20260926 09:32:49.078765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:49.163877 4534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:49.164155 4534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:49.164196 4534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:49.165500 4534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:49.165607 4534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:49.165663 4534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:49.167498 4534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:49.167804 4534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:49.168064 4534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:49.173907 4540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:49.174005 4534 server_base.cc:1034] running on GCE node
W20260926 09:32:49.173952 4542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:49.173943 4539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:49.174374 4534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:49.174676 4534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:49.175866 4534 hybrid_clock.cc:648] HybridClock initialized: now 1790415169175846 us; error 31 us; skew 500 ppm
Sep 26 09:32:49 dist-test-slave-21h8 krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:49.178452 4534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:49.178853 4534 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:49.178930 4534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:49.179085 4534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:49.180271 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "a7bc9acb839b43f2acc369f4ee8600fe"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "25d9cdace3c564faf464036ced26b801"
server_key_iv: "b3c921eed95cfeb944fece84ba61b866"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.180658 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a7bc9acb839b43f2acc369f4ee8600fe"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "25d9cdace3c564faf464036ced26b801"
server_key_iv: "b3c921eed95cfeb944fece84ba61b866"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.182077 4534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:49.182847 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.183076 4534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:49.183203 4534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "a7bc9acb839b43f2acc369f4ee8600fe"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "25d9cdace3c564faf464036ced26b801"
server_key_iv: "b3c921eed95cfeb944fece84ba61b866"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.183285 4534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:49.204185 4534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:49.206267 4534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:49.206460 4534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:49.206707 4534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:49.207060 4534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:49.207129 4534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.207165 4534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:49.207191 4534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.213255 4534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38169
I20260926 09:32:49.213291 4661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38169 every 8 connection(s)
I20260926 09:32:49.213578 4534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:32:49.213753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 4534
I20260926 09:32:49.213873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:49.214107 392 external_mini_cluster.cc:1428] Setting key 0ff3e786c9ef4ed0de4e2946c70c922b
Sep 26 09:32:49 dist-test-slave-21h8 krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:32:49.225039 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.214596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46337 (local address 127.0.98.62:37431)
0926 09:32:49.214728 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.214730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.215023 (+ 293us) server_negotiation.cc:407] Connection header received
0926 09:32:49.215566 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.215567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.215582 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.215657 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:49.217032 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.217534 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.219141 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.219406 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.219474 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.221985 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:49.221988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:49.221988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:49.221996 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:49.223770 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:49.224184 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:49.224185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:49.224185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:49.224240 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:49.224464 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:49.224465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:49.224465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:49.224574 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:49.224628 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.224894 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.224963 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260926 09:32:49.225274 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.214593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:46337)
0926 09:32:49.214846 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.214891 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.215461 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.215766 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.215771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.216126 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:49.216920 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.216927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.217627 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.217628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.219030 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.219033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.219506 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.219508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.219674 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.220234 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:49.220246 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:49.221879 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:49.223856 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:49.223862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:49.223865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:49.224109 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:49.224304 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:49.224305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:49.224305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:49.224406 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:49.224641 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:49.224647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:49.224780 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.224863 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.224989 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260926 09:32:49.225881 4662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260926 09:32:49.225986 4662 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:49.226150 4662 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 09:32:49.226752 4472 ts_manager.cc:194] Registered new tserver with Master: a7bc9acb839b43f2acc369f4ee8600fe (127.0.98.1:38169)
I20260926 09:32:49.227705 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:49.243533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:49.328459 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:49.328706 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:49.328748 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:49.329999 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:49.330075 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:49.330123 4668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:49.331863 4668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:49.332149 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:49.332389 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:49.337661 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:49.337639 4673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:49.337886 4668 server_base.cc:1034] running on GCE node
W20260926 09:32:49.337980 4674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:49.338351 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:49.338685 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:49.339826 4668 hybrid_clock.cc:648] HybridClock initialized: now 1790415169339789 us; error 60 us; skew 500 ppm
Sep 26 09:32:49 dist-test-slave-21h8 krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:49.342649 4668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:49.343079 4668 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:49.343164 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:49.343338 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:49.344874 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "ebf66b4af12e41fc9093b9be978bb1c7"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "fdcf82ab5ad5d4f8b4c06d0c5560e0d0"
server_key_iv: "979ef15198e35dd0413c02ef1bd2cda3"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.345343 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ebf66b4af12e41fc9093b9be978bb1c7"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "fdcf82ab5ad5d4f8b4c06d0c5560e0d0"
server_key_iv: "979ef15198e35dd0413c02ef1bd2cda3"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.347043 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:49.348090 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.348263 4668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:49.348359 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "ebf66b4af12e41fc9093b9be978bb1c7"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "fdcf82ab5ad5d4f8b4c06d0c5560e0d0"
server_key_iv: "979ef15198e35dd0413c02ef1bd2cda3"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.348433 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:49.378109 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:49.381155 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:49.381322 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:49.381664 4668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:49.382071 4668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:49.382119 4668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.382155 4668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:49.382184 4668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.389685 4668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37123
I20260926 09:32:49.389722 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37123 every 8 connection(s)
I20260926 09:32:49.390049 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:32:49 dist-test-slave-21h8 krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:49.399375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 4668
I20260926 09:32:49.399485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:49.399683 392 external_mini_cluster.cc:1428] Setting key d7e5a88170fffed29eea47267f4acafa
I20260926 09:32:49.399962 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.390865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37115 (local address 127.0.98.62:37431)
0926 09:32:49.390973 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.390975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.391166 (+ 191us) server_negotiation.cc:407] Connection header received
0926 09:32:49.391753 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.391754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.391764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.391832 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:49.393041 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.393557 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.394536 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.394728 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.394779 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.397014 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:49.397016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:49.397016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:49.397023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:49.398475 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:49.398849 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:49.398850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:49.398850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:49.398903 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:49.399168 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:49.399169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:49.399169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:49.399272 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:49.399385 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.399710 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.399847 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 09:32:49.400144 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.390830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:37115)
0926 09:32:49.391004 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.391055 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.391633 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.391926 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.391930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.392405 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:49.392917 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.392923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.393649 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.393650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.394457 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.394459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.394805 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.394806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.394910 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.395538 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:49.395547 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:49.396925 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:49.398558 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:49.398564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:49.398566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:49.398778 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:49.398967 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:49.398967 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:49.398967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:49.399082 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:49.399362 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:49.399369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:49.399571 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.399694 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.399893 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260926 09:32:49.400655 4796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260926 09:32:49.400780 4796 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:49.400931 4796 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260926 09:32:49.401348 4472 ts_manager.cc:194] Registered new tserver with Master: ebf66b4af12e41fc9093b9be978bb1c7 (127.0.98.2:37123)
I20260926 09:32:49.401736 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37115
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:49.426831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:49.511873 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:49.512100 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:49.512127 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:49.513304 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:49.513340 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:49.513367 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:49.514864 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:49.515067 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:49.515242 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:49.520399 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:49.520434 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:49.520407 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:49.520676 4801 server_base.cc:1034] running on GCE node
I20260926 09:32:49.520980 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:49.521287 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:49.522481 4801 hybrid_clock.cc:648] HybridClock initialized: now 1790415169522455 us; error 40 us; skew 500 ppm
Sep 26 09:32:49 dist-test-slave-21h8 krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:49.525295 4801 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:49.525602 4801 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:49.525648 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:49.525741 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:49.526790 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "50ab898137f54ab2a1809f33609c2eb5"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "795d5ca1273f22debfe51e10516e2ae6"
server_key_iv: "d2ce2777e408a4e9f3058a81f5cc8304"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.527114 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "50ab898137f54ab2a1809f33609c2eb5"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "795d5ca1273f22debfe51e10516e2ae6"
server_key_iv: "d2ce2777e408a4e9f3058a81f5cc8304"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.528473 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:32:49.529172 4815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.529351 4801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:32:49.529438 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "50ab898137f54ab2a1809f33609c2eb5"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "795d5ca1273f22debfe51e10516e2ae6"
server_key_iv: "d2ce2777e408a4e9f3058a81f5cc8304"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.529495 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:49.541276 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:49.545303 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:49.545467 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:49.545719 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:49.546093 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:49.546254 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.546324 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:49.546379 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.553738 4801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33673
I20260926 09:32:49.553804 4928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33673 every 8 connection(s)
I20260926 09:32:49.554069 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:32:49 dist-test-slave-21h8 krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:49.561991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 4801
I20260926 09:32:49.562119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:49.562304 392 external_mini_cluster.cc:1428] Setting key 5377768b0d1508f495cf343a7b4400cc
I20260926 09:32:49.564123 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.554831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59421 (local address 127.0.98.62:37431)
0926 09:32:49.554911 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.554914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.555112 (+ 198us) server_negotiation.cc:407] Connection header received
0926 09:32:49.555769 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.555770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.555784 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.555835 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:49.556803 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.557322 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.558354 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.558567 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.558649 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.560882 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:49.560883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:49.560883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:49.560891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:49.562477 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:49.563092 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:49.563093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:49.563093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:49.563160 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:49.563553 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:49.563554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:49.563554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:49.563669 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:49.563725 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.563986 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.564054 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:32:49.564311 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.554803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:59421)
0926 09:32:49.554974 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.555019 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.555648 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.555906 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.555910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.556188 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:49.556693 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.556698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.557423 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.557425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.558244 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.558246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.558654 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.558655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.558792 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.559356 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:49.559367 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:49.560771 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:49.562632 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:49.562638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:49.562640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:49.563002 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:49.563255 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:49.563256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:49.563257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:49.563430 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:49.563732 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:49.563742 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:49.563864 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.563943 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.564067 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260926 09:32:49.564846 4929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260926 09:32:49.565014 4929 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:49.565197 4929 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260926 09:32:49.565619 4472 ts_manager.cc:194] Registered new tserver with Master: 50ab898137f54ab2a1809f33609c2eb5 (127.0.98.3:33673)
I20260926 09:32:49.566020 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59421
I20260926 09:32:49.574659 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:32:49.582046 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.576099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.98.62:37431)
0926 09:32:49.576142 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.576144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.576216 (+ 72us) server_negotiation.cc:407] Connection header received
0926 09:32:49.576272 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.576273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.576281 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.576320 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:49.576918 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.577405 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.578099 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.578272 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.578323 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.579555 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:49.579557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:49.579557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:49.579564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:49.580837 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:49.581198 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:49.581199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:49.581200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:49.581260 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:49.581482 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:49.581482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:49.581483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:49.581636 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:49.581753 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.581865 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.581941 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:32:49.583571 4472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:49.585106 4472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:49.591827 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.587635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38169 (local address 127.0.0.1:53294)
0926 09:32:49.587793 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.587833 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.587893 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.588200 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.588203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.588205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:49.588275 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.588277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.589376 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.589377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.590472 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.590475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.591477 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.591478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.591564 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.591578 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.591692 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.591750 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":110,"threads_started":1}
I20260926 09:32:49.591840 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.587741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.0.98.1:38169)
0926 09:32:49.587914 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.587919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.587931 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:32:49.587977 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.587980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.587998 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.588094 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:49.588358 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.589272 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.590584 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.591399 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.591465 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.591530 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.591710 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.591750 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260926 09:32:49.592286 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.587883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37123 (local address 127.0.0.1:46864)
0926 09:32:49.588021 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.588032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.588164 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.588543 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.588543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.588545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:49.588600 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.588602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.589979 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.589980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.591140 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.591142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.592047 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.592048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.592113 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.592117 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.592150 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.592196 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":26,"thread_start_us":74,"threads_started":1}
I20260926 09:32:49.592288 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.588044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33673 (local address 127.0.0.1:60062)
0926 09:32:49.588449 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.588461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.588508 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.588703 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.588703 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.588706 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:49.588745 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.588746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.589982 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.589983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.590977 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.590979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.592043 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.592044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.592136 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.592140 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.592187 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.592215 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":24,"thread_start_us":68,"threads_started":1}
I20260926 09:32:49.592480 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.588149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.0.98.3:33673)
0926 09:32:49.588306 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.588311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.588567 (+ 256us) server_negotiation.cc:407] Connection header received
0926 09:32:49.588612 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.588615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.588631 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.588704 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:49.588803 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.589863 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.591080 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.591882 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.591954 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.592009 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.592222 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.592267 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260926 09:32:49.592754 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.588235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.0.98.2:37123)
0926 09:32:49.588365 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.588370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.588381 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:32:49.588419 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.588421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.588437 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.588677 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:49.588760 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.589873 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.591233 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.591959 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.592515 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.592569 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.592620 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.592671 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260926 09:32:49.592746 4596 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcb51dd984f408993edacce51f8de14 (DEFAULT_TABLE table=test-table [id=0fc5153a5b9e4ebc974211e136da4c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:49.593134 4596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcb51dd984f408993edacce51f8de14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:49.593206 4863 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcb51dd984f408993edacce51f8de14 (DEFAULT_TABLE table=test-table [id=0fc5153a5b9e4ebc974211e136da4c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:49.593326 4730 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcb51dd984f408993edacce51f8de14 (DEFAULT_TABLE table=test-table [id=0fc5153a5b9e4ebc974211e136da4c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:49.593490 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcb51dd984f408993edacce51f8de14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:49.593565 4730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcb51dd984f408993edacce51f8de14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:49.596247 4946 tablet_bootstrap.cc:492] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Bootstrap starting.
I20260926 09:32:49.596900 4946 tablet_bootstrap.cc:654] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:49.596922 4947 tablet_bootstrap.cc:492] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe: Bootstrap starting.
I20260926 09:32:49.597426 4946 log.cc:824] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:49.597462 4947 tablet_bootstrap.cc:654] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:49.598075 4947 log.cc:824] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:49.598232 4946 tablet_bootstrap.cc:492] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: No bootstrap required, opened a new log
I20260926 09:32:49.598305 4946 ts_tablet_manager.cc:1397] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:49.598448 4948 tablet_bootstrap.cc:492] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: Bootstrap starting.
I20260926 09:32:49.598855 4947 tablet_bootstrap.cc:492] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe: No bootstrap required, opened a new log
I20260926 09:32:49.598922 4947 ts_tablet_manager.cc:1397] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:49.599035 4948 tablet_bootstrap.cc:654] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:49.599620 4948 log.cc:824] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:49.600308 4948 tablet_bootstrap.cc:492] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: No bootstrap required, opened a new log
I20260926 09:32:49.600392 4948 ts_tablet_manager.cc:1397] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:49.600417 4946 raft_consensus.cc:348] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.600625 4946 raft_consensus.cc:374] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:49.600675 4946 raft_consensus.cc:729] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf66b4af12e41fc9093b9be978bb1c7, State: Initialized, Role: FOLLOWER
I20260926 09:32:49.600793 4946 consensus_queue.cc:260] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.600930 4947 raft_consensus.cc:348] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.601097 4947 raft_consensus.cc:374] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:49.601094 4796 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260926 09:32:49.601141 4947 raft_consensus.cc:729] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7bc9acb839b43f2acc369f4ee8600fe, State: Initialized, Role: FOLLOWER
I20260926 09:32:49.601281 4947 consensus_queue.cc:260] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.601091 4946 ts_tablet_manager.cc:1428] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:32:49.601562 4947 ts_tablet_manager.cc:1428] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:32:49.601672 4662 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260926 09:32:49.602569 4948 raft_consensus.cc:348] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.602720 4948 raft_consensus.cc:374] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:49.602766 4948 raft_consensus.cc:729] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ab898137f54ab2a1809f33609c2eb5, State: Initialized, Role: FOLLOWER
I20260926 09:32:49.602876 4948 consensus_queue.cc:260] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.603156 4948 ts_tablet_manager.cc:1428] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:32:49.603250 4929 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
W20260926 09:32:49.640574 4797 tablet.cc:2369] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:32:49.644652 4953 raft_consensus.cc:482] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:49.644770 4953 raft_consensus.cc:504] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.645072 4953 leader_election.cc:288] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebf66b4af12e41fc9093b9be978bb1c7 (127.0.98.2:37123), 50ab898137f54ab2a1809f33609c2eb5 (127.0.98.3:33673)
I20260926 09:32:49.648380 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.645439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50807 (local address 127.0.98.2:37123)
0926 09:32:49.645522 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.645524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.645534 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:49.645565 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.645566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.645579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.645671 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:49.645839 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.646329 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.647633 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.648007 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.648098 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.648109 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.648261 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.648297 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 09:32:49.648394 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.645468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45729 (local address 127.0.98.3:33673)
0926 09:32:49.645514 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.645516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.645688 (+ 172us) server_negotiation.cc:407] Connection header received
0926 09:32:49.645760 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.645760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.645772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.645843 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:49.645996 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.646451 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.647645 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.648023 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.648113 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.648122 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.648291 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.648326 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:32:49.648419 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.645286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37123 (local address 127.0.98.1:50807)
0926 09:32:49.645351 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.645366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.645434 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.645678 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.645679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.645681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:49.645754 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.645756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.646425 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.646426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.647536 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.647538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.648099 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.648099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.648176 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.648179 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.648256 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.648297 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 09:32:49.648420 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.645394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33673 (local address 127.0.98.1:45729)
0926 09:32:49.645611 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.645619 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.645694 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.645846 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.645846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.645848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:49.645924 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.645925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.646542 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.646543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.647544 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:49.647547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.648112 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.648113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:49.648198 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.648202 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.648275 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.648315 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":6,"thread_start_us":147,"threads_started":1}
I20260926 09:32:49.648877 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcb51dd984f408993edacce51f8de14" candidate_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ab898137f54ab2a1809f33609c2eb5" is_pre_election: true
I20260926 09:32:49.648890 4750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcb51dd984f408993edacce51f8de14" candidate_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" is_pre_election: true
I20260926 09:32:49.649065 4883 raft_consensus.cc:2454] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7bc9acb839b43f2acc369f4ee8600fe in term 0.
I20260926 09:32:49.649087 4750 raft_consensus.cc:2454] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7bc9acb839b43f2acc369f4ee8600fe in term 0.
I20260926 09:32:49.649327 4552 leader_election.cc:302] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7bc9acb839b43f2acc369f4ee8600fe, ebf66b4af12e41fc9093b9be978bb1c7; no voters:
I20260926 09:32:49.649477 4953 raft_consensus.cc:2781] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:49.649536 4953 raft_consensus.cc:482] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:49.649561 4953 raft_consensus.cc:3037] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:49.650251 4953 raft_consensus.cc:504] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.650419 4953 leader_election.cc:288] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [CANDIDATE]: Term 1 election: Requested vote from peers ebf66b4af12e41fc9093b9be978bb1c7 (127.0.98.2:37123), 50ab898137f54ab2a1809f33609c2eb5 (127.0.98.3:33673)
I20260926 09:32:49.650611 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcb51dd984f408993edacce51f8de14" candidate_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ab898137f54ab2a1809f33609c2eb5"
I20260926 09:32:49.650609 4750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcb51dd984f408993edacce51f8de14" candidate_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf66b4af12e41fc9093b9be978bb1c7"
I20260926 09:32:49.650713 4750 raft_consensus.cc:3037] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:49.650707 4883 raft_consensus.cc:3037] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:49.651576 4883 raft_consensus.cc:2454] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7bc9acb839b43f2acc369f4ee8600fe in term 1.
I20260926 09:32:49.651676 4750 raft_consensus.cc:2454] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7bc9acb839b43f2acc369f4ee8600fe in term 1.
I20260926 09:32:49.651732 4552 leader_election.cc:302] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ab898137f54ab2a1809f33609c2eb5, a7bc9acb839b43f2acc369f4ee8600fe; no voters:
I20260926 09:32:49.651865 4953 raft_consensus.cc:2781] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:49.652024 4953 raft_consensus.cc:686] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 1 LEADER]: Becoming Leader. State: Replica: a7bc9acb839b43f2acc369f4ee8600fe, State: Running, Role: LEADER
I20260926 09:32:49.652172 4953 consensus_queue.cc:237] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } }
I20260926 09:32:49.653098 4471 catalog_manager.cc:5495] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe reported cstate change: term changed from 0 to 1, leader changed from <none> to a7bc9acb839b43f2acc369f4ee8600fe (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc9acb839b43f2acc369f4ee8600fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 } health_report { overall_health: UNKNOWN } } }
I20260926 09:32:49.666646 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.664407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.0.98.1:38169)
0926 09:32:49.664455 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.664456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.664465 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:49.664527 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.664528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.664553 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.664613 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:49.664757 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.665208 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.665945 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:49.666124 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:49.666175 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:49.666298 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:49.666403 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:49.666460 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.666491 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.666508 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:32:49.668635 4750 raft_consensus.cc:1261] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 1 FOLLOWER]: Refusing update from remote peer a7bc9acb839b43f2acc369f4ee8600fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:49.668691 4883 raft_consensus.cc:1261] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 1 FOLLOWER]: Refusing update from remote peer a7bc9acb839b43f2acc369f4ee8600fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:49.669066 4956 consensus_queue.cc:1035] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf66b4af12e41fc9093b9be978bb1c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:49.669247 4953 consensus_queue.cc:1035] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ab898137f54ab2a1809f33609c2eb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:49.671496 4959 mvcc.cc:205] Tried to move back new op lower bound from 7333540534959771648 to 7333540534896095232. Current Snapshot: MvccSnapshot[applied={T|T < 7333540534959771648}]
I20260926 09:32:49.672355 4960 mvcc.cc:205] Tried to move back new op lower bound from 7333540534959771648 to 7333540534896095232. Current Snapshot: MvccSnapshot[applied={T|T < 7333540534959771648}]
I20260926 09:32:49.673849 4472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:49.673964 4472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:49.674767 4472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ea105210bb24c8caf61ae68b1710dfc: Sending DeleteTablet for 3 replicas of tablet 0dcb51dd984f408993edacce51f8de14
I20260926 09:32:49.675060 4596 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcb51dd984f408993edacce51f8de14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53294
I20260926 09:32:49.675112 4730 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcb51dd984f408993edacce51f8de14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46864
I20260926 09:32:49.675143 4863 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcb51dd984f408993edacce51f8de14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60062
I20260926 09:32:49.675386 4969 tablet_replica.cc:330] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: stopping tablet replica
I20260926 09:32:49.675407 4968 tablet_replica.cc:330] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe: stopping tablet replica
I20260926 09:32:49.675484 4969 raft_consensus.cc:2229] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:49.675485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 4534
I20260926 09:32:49.675513 4968 raft_consensus.cc:2229] T 0dcb51dd984f408993edacce51f8de14 P a7bc9acb839b43f2acc369f4ee8600fe [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:49.675590 4969 raft_consensus.cc:2258] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:49.675740 4970 tablet_replica.cc:330] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: stopping tablet replica
I20260926 09:32:49.675836 4970 raft_consensus.cc:2229] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:49.675940 4970 raft_consensus.cc:2258] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:49.676429 4970 ts_tablet_manager.cc:1905] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:49.677814 4969 ts_tablet_manager.cc:1905] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:49.678218 4970 ts_tablet_manager.cc:1918] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:49.678295 4970 log.cc:1196] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/0dcb51dd984f408993edacce51f8de14
I20260926 09:32:49.678566 4970 ts_tablet_manager.cc:1939] T 0dcb51dd984f408993edacce51f8de14 P 50ab898137f54ab2a1809f33609c2eb5: Deleting consensus metadata
I20260926 09:32:49.679040 4459 catalog_manager.cc:4841] TS 50ab898137f54ab2a1809f33609c2eb5 (127.0.98.3:33673): tablet 0dcb51dd984f408993edacce51f8de14 (table test-table [id=0fc5153a5b9e4ebc974211e136da4c6f]) successfully deleted
I20260926 09:32:49.679466 4969 ts_tablet_manager.cc:1918] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:49.679522 4969 log.cc:1196] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/0dcb51dd984f408993edacce51f8de14
I20260926 09:32:49.679831 4969 ts_tablet_manager.cc:1939] T 0dcb51dd984f408993edacce51f8de14 P ebf66b4af12e41fc9093b9be978bb1c7: Deleting consensus metadata
I20260926 09:32:49.680248 4459 catalog_manager.cc:4841] TS ebf66b4af12e41fc9093b9be978bb1c7 (127.0.98.2:37123): tablet 0dcb51dd984f408993edacce51f8de14 (table test-table [id=0fc5153a5b9e4ebc974211e136da4c6f]) successfully deleted
W20260926 09:32:49.681205 4459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38169 (error 108)
I20260926 09:32:49.681484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 4668
I20260926 09:32:49.682240 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.681619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38169 (local address 127.0.0.1:53316)
0926 09:32:49.682163 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.682212 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
W20260926 09:32:49.682425 4459 catalog_manager.cc:4584] TS a7bc9acb839b43f2acc369f4ee8600fe (127.0.98.1:38169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0dcb51dd984f408993edacce51f8de14: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38169: connect: Connection refused (error 111)
I20260926 09:32:49.687109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 4801
I20260926 09:32:49.691828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 4441
2026-09-26T09:32:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2429 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T09:32:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:49Z Disabled control of system clock
I20260926 09:32:49.719575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41873
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:32:49.800505 4975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:49.800722 4975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:49.800743 4975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:49.801826 4975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:49.801860 4975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:49.801873 4975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:49.801885 4975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:49.801898 4975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:49.803174 4975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:49.803474 4975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:49.803730 4975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:49.808521 4981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:49.808560 4983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:49.808540 4980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:49.808759 4975 server_base.cc:1034] running on GCE node
I20260926 09:32:49.809003 4975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:49.809357 4975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:49.810535 4975 hybrid_clock.cc:648] HybridClock initialized: now 1790415169810514 us; error 42 us; skew 500 ppm
I20260926 09:32:49.811896 4975 webserver.cc:460] Webserver started at http://127.0.98.62:41221/ using document root <none> and password file <none>
I20260926 09:32:49.812170 4975 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:49.812247 4975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:49.812398 4975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:49.813535 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "57c89ee98f434eacb15ba3c4c156fe2c"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "752f917c92ebcfe1626e8cc1b8133610"
server_key_iv: "8adc85392b17baad800511e1ea5d52f7"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.813910 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "57c89ee98f434eacb15ba3c4c156fe2c"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "752f917c92ebcfe1626e8cc1b8133610"
server_key_iv: "8adc85392b17baad800511e1ea5d52f7"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.815227 4975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:49.816017 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.816190 4975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:49.816293 4975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "57c89ee98f434eacb15ba3c4c156fe2c"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "752f917c92ebcfe1626e8cc1b8133610"
server_key_iv: "8adc85392b17baad800511e1ea5d52f7"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.816375 4975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:49.819830 4975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:49.820104 4975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:49.820256 4975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:49.824056 4975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41873
I20260926 09:32:49.824080 5041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41873 every 8 connection(s)
I20260926 09:32:49.824414 4975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:49.825037 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:49.827800 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c: Bootstrap starting.
I20260926 09:32:49.828336 5042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:49.828573 5042 log.cc:824] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:49.829208 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c: No bootstrap required, opened a new log
I20260926 09:32:49.830689 5042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c89ee98f434eacb15ba3c4c156fe2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } }
I20260926 09:32:49.830825 5042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:49.830847 5042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c89ee98f434eacb15ba3c4c156fe2c, State: Initialized, Role: FOLLOWER
I20260926 09:32:49.830998 5042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c89ee98f434eacb15ba3c4c156fe2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } }
I20260926 09:32:49.831082 5042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:49.831112 5042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:49.831141 5042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:49.831741 5042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c89ee98f434eacb15ba3c4c156fe2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } }
I20260926 09:32:49.831826 5042 leader_election.cc:302] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57c89ee98f434eacb15ba3c4c156fe2c; no voters:
I20260926 09:32:49.831950 5042 leader_election.cc:288] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:49.832042 5045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:49.832211 5045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [term 1 LEADER]: Becoming Leader. State: Replica: 57c89ee98f434eacb15ba3c4c156fe2c, State: Running, Role: LEADER
I20260926 09:32:49.832242 5042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:49.832300 5045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c89ee98f434eacb15ba3c4c156fe2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } }
I20260926 09:32:49.832674 5046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57c89ee98f434eacb15ba3c4c156fe2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c89ee98f434eacb15ba3c4c156fe2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } } }
I20260926 09:32:49.832767 5046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:49.832945 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57c89ee98f434eacb15ba3c4c156fe2c. Latest consensus state: current_term: 1 leader_uuid: "57c89ee98f434eacb15ba3c4c156fe2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c89ee98f434eacb15ba3c4c156fe2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41873 } } }
I20260926 09:32:49.833011 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:49.832998 5053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:49.833572 5053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:49.833627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 4975
I20260926 09:32:49.833714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:49.833896 392 external_mini_cluster.cc:1428] Setting key 5f05bb56b8c1e5cb4844a6eb92391c3a
I20260926 09:32:49.835583 5053 catalog_manager.cc:1347] Generated new cluster ID: 396b2f210dd24486ba5aedbe34547d00
I20260926 09:32:49.835638 5053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:49.840183 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.834562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.0.98.62:41873)
0926 09:32:49.834761 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.834771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.834806 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:32:49.834988 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.834996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.835039 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.835189 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:49.839657 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:49.839661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:49.839662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:49.839667 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:49.839724 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:49.839744 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:49.839828 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.839908 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.839978 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260926 09:32:49.847669 5053 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:49.847856 5053 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:49.854714 5053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c: Generated new TSK 0
I20260926 09:32:49.854933 5053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:32:49.861423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41873
--builtin_ntp_servers=127.0.98.20:34451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:32:49.944669 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:49.944890 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:49.944913 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:49.946245 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:49.946286 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:49.946313 5066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:49.947732 5066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:49.948002 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:49.948323 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:49.954466 5072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:49.954721 5071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:49.955242 5074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:49.955655 5066 server_base.cc:1034] running on GCE node
I20260926 09:32:49.955902 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:49.956194 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:49.957355 5066 hybrid_clock.cc:648] HybridClock initialized: now 1790415169957341 us; error 25 us; skew 500 ppm
I20260926 09:32:49.958755 5066 webserver.cc:460] Webserver started at http://127.0.98.1:44665/ using document root <none> and password file <none>
I20260926 09:32:49.959013 5066 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:49.959092 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:49.959261 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:49.960433 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "1f7ce94c0005475d97c4b467ae992e5b"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "dc85f47a281012980cb7b50bc0d8117b"
server_key_iv: "d2de8db8f84bb7ad82b5aaf151720fcf"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.960834 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1f7ce94c0005475d97c4b467ae992e5b"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "dc85f47a281012980cb7b50bc0d8117b"
server_key_iv: "d2de8db8f84bb7ad82b5aaf151720fcf"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.962299 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:49.963204 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.963441 5066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:49.963532 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "1f7ce94c0005475d97c4b467ae992e5b"
format_stamp: "Formatted at 2026-09-26 09:32:49 on dist-test-slave-21h8"
server_key: "dc85f47a281012980cb7b50bc0d8117b"
server_key_iv: "d2de8db8f84bb7ad82b5aaf151720fcf"
server_key_version: "encryptionkey@0"
I20260926 09:32:49.963603 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:49.968443 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:49.968753 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:49.968895 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:49.969127 5066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:49.969514 5066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:49.969568 5066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.969605 5066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:49.969627 5066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:49.975608 5066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44967
I20260926 09:32:49.975692 5193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44967 every 8 connection(s)
I20260926 09:32:49.976051 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:32:49.976617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 5066
I20260926 09:32:49.976719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:49.976974 392 external_mini_cluster.cc:1428] Setting key f6afde50023a38b2269d9f21eaf23b51
I20260926 09:32:49.978091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41873
--builtin_ntp_servers=127.0.98.20:34451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 09:32:49.980013 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.976927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59043 (local address 127.0.98.62:41873)
0926 09:32:49.976979 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:49.976982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:49.977281 (+ 299us) server_negotiation.cc:407] Connection header received
0926 09:32:49.978099 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:49.978100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:49.978105 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:49.978224 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:49.979754 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:49.979756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:49.979756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:49.979761 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:49.979777 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:49.979781 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:49.979841 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:49.979928 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 09:32:49.979944 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:32:49.980275 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:49.976924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41873 (local address 127.0.98.1:59043)
0926 09:32:49.977109 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:49.977157 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:49.977828 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:49.978334 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:49.978339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:49.978348 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:49.979560 (+ 1212us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:32:49.979560 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:49.979603 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:32:49.979638 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:49.979842 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:49.979847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:49.979850 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:49.979940 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:32:49.980005 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260926 09:32:49.980862 5194 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41873
I20260926 09:32:49.980964 5194 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:49.981122 5194 heartbeater.cc:511] Master 127.0.98.62:41873 requested a full tablet report, sending...
I20260926 09:32:49.981674 5006 ts_manager.cc:194] Registered new tserver with Master: 1f7ce94c0005475d97c4b467ae992e5b (127.0.98.1:44967)
W20260926 09:32:50.060367 5197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:50.060662 5197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:50.060702 5197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:50.061961 5197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:50.062045 5197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:50.062091 5197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:50.063522 5197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:50.063823 5197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:50.064160 5197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:50.069538 5203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:50.069509 5205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:50.069515 5202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:50.069813 5197 server_base.cc:1034] running on GCE node
I20260926 09:32:50.070050 5197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:50.070304 5197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:50.071455 5197 hybrid_clock.cc:648] HybridClock initialized: now 1790415170071442 us; error 27 us; skew 500 ppm
I20260926 09:32:50.072729 5197 webserver.cc:460] Webserver started at http://127.0.98.2:45089/ using document root <none> and password file <none>
I20260926 09:32:50.072952 5197 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:50.072994 5197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:50.073086 5197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:50.074175 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "0cc416986f304fd981fa9c53c22ee4e3"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "3325b919160e8e900d1aeb677e116f34"
server_key_iv: "691e64b95aee380ed6665673310f2d2d"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.074479 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0cc416986f304fd981fa9c53c22ee4e3"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "3325b919160e8e900d1aeb677e116f34"
server_key_iv: "691e64b95aee380ed6665673310f2d2d"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.075851 5197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:32:50.076565 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:50.076728 5197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:32:50.076797 5197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "0cc416986f304fd981fa9c53c22ee4e3"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "3325b919160e8e900d1aeb677e116f34"
server_key_iv: "691e64b95aee380ed6665673310f2d2d"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.076849 5197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:50.080124 5197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:50.080343 5197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:50.080442 5197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:50.080595 5197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:50.080832 5197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:50.080861 5197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:50.080884 5197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:50.080897 5197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:50.086419 5197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37845
I20260926 09:32:50.086512 5324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37845 every 8 connection(s)
I20260926 09:32:50.086745 5197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:50.089798 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.087483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35121 (local address 127.0.98.62:41873)
0926 09:32:50.087561 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.087563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.087838 (+ 275us) server_negotiation.cc:407] Connection header received
0926 09:32:50.088354 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.088356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.088362 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.088411 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.089536 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.089538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.089539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.089543 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.089560 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.089564 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.089619 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.089710 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.089725 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:32:50.090003 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.087487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41873 (local address 127.0.98.2:35121)
0926 09:32:50.087711 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.087746 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.088243 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.088488 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.088491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.088501 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:50.089378 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:32:50.089378 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:50.089407 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:32:50.089435 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:50.089614 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:50.089619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:50.089622 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:50.089722 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 09:32:50.089783 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260926 09:32:50.090531 5325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41873
I20260926 09:32:50.090608 5325 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:50.090788 5325 heartbeater.cc:511] Master 127.0.98.62:41873 requested a full tablet report, sending...
I20260926 09:32:50.091251 5006 ts_manager.cc:194] Registered new tserver with Master: 0cc416986f304fd981fa9c53c22ee4e3 (127.0.98.2:37845)
I20260926 09:32:50.093657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 5197
I20260926 09:32:50.093741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:50.093957 392 external_mini_cluster.cc:1428] Setting key 190f93333c24a4ba2730c14d543b451e
I20260926 09:32:50.094801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41873
--builtin_ntp_servers=127.0.98.20:34451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:32:50.176716 5328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:50.176924 5328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:50.176946 5328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:50.178145 5328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:50.178188 5328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:50.178213 5328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:50.179749 5328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:50.180007 5328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:50.180364 5328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:50.185345 5336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:50.185354 5334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:50.185487 5328 server_base.cc:1034] running on GCE node
W20260926 09:32:50.185364 5333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:50.185814 5328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:50.186107 5328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:50.187279 5328 hybrid_clock.cc:648] HybridClock initialized: now 1790415170187256 us; error 36 us; skew 500 ppm
I20260926 09:32:50.188525 5328 webserver.cc:460] Webserver started at http://127.0.98.3:33799/ using document root <none> and password file <none>
I20260926 09:32:50.188809 5328 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:50.188908 5328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:50.189061 5328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:50.190141 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "ef07a3cd350344bb84f34e7b057bcaea"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "e223dc47f561d09c0f7f687dd99ab6d6"
server_key_iv: "ed22a32d74a9bdda4160813e3dc2e170"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.190508 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ef07a3cd350344bb84f34e7b057bcaea"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "e223dc47f561d09c0f7f687dd99ab6d6"
server_key_iv: "ed22a32d74a9bdda4160813e3dc2e170"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.192215 5328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 09:32:50.193042 5342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:50.193233 5328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:50.193336 5328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "ef07a3cd350344bb84f34e7b057bcaea"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "e223dc47f561d09c0f7f687dd99ab6d6"
server_key_iv: "ed22a32d74a9bdda4160813e3dc2e170"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.193420 5328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:50.196823 5328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:50.197093 5328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:50.197237 5328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:50.197440 5328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:50.197753 5328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:50.197811 5328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:50.197850 5328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:50.197878 5328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:50.203598 5328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34889
I20260926 09:32:50.203681 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34889 every 8 connection(s)
I20260926 09:32:50.203989 5328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:32:50.207561 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.204749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35371 (local address 127.0.98.62:41873)
0926 09:32:50.204793 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.204795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.205039 (+ 244us) server_negotiation.cc:407] Connection header received
0926 09:32:50.205593 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.205594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.205601 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.205671 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.207254 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.207256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.207256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.207261 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.207279 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.207283 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.207357 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.207465 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.207483 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:32:50.207808 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.204692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41873 (local address 127.0.98.3:35371)
0926 09:32:50.204881 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.204923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.205487 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.205844 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.205848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.205865 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:50.207056 (+ 1191us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:32:50.207057 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:50.207096 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:32:50.207124 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:50.207357 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:50.207363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:50.207366 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:50.207476 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 09:32:50.207571 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260926 09:32:50.208390 5456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41873
I20260926 09:32:50.208489 5456 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:50.208598 5456 heartbeater.cc:511] Master 127.0.98.62:41873 requested a full tablet report, sending...
I20260926 09:32:50.208992 5006 ts_manager.cc:194] Registered new tserver with Master: ef07a3cd350344bb84f34e7b057bcaea (127.0.98.3:34889)
I20260926 09:32:50.209333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 5328
I20260926 09:32:50.209419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:50.209645 392 external_mini_cluster.cc:1428] Setting key c809f66ddf4bfab625554257f3b09cfc
I20260926 09:32:50.210409 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:32:50.212868 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.211886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.0.98.62:41873)
0926 09:32:50.211965 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.211967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.212007 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:32:50.212061 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.212061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.212067 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.212110 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.212647 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.212649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.212649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.212652 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.212665 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.212668 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.212712 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.212744 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.212758 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 09:32:50.214452 5006 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:50.216174 5006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:50.220367 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.218610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.0.98.1:44967)
0926 09:32:50.218832 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.218837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.218940 (+ 103us) server_negotiation.cc:407] Connection header received
0926 09:32:50.218993 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.218995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.219007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.219090 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.219979 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.219985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.219985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.219989 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.220018 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.220037 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.220087 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.220240 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.220259 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260926 09:32:50.220428 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.218418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37845 (local address 127.0.0.1:35336)
0926 09:32:50.219002 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.219011 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.219041 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.219177 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.219181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.219188 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:50.219718 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:32:50.219718 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:50.219735 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:32:50.219756 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:50.220300 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:50.220300 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:50.220301 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:32:50.220342 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 09:32:50.220355 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":103,"threads_started":1}
I20260926 09:32:50.220548 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.218405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44967 (local address 127.0.0.1:53624)
0926 09:32:50.218880 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.218888 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.218933 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.219510 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.219511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.219514 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:50.219869 (+ 355us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:32:50.219870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:50.219877 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:32:50.219889 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:50.220112 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:50.220118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:50.220121 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:50.220461 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 09:32:50.220474 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260926 09:32:50.220693 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.218678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.0.98.2:37845)
0926 09:32:50.219033 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.219047 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.219059 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:32:50.219096 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.219099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.219107 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.219176 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.220163 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.220167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.220168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.220171 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.220204 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.220220 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.220277 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.220593 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.220609 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260926 09:32:50.220698 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.218609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.0.98.3:34889)
0926 09:32:50.218776 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.218780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.218895 (+ 115us) server_negotiation.cc:407] Connection header received
0926 09:32:50.219106 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.219112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.219123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.219188 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.220234 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.220241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.220241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.220245 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.220275 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.220297 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.220380 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.220509 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.220530 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260926 09:32:50.221042 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.218405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34889 (local address 127.0.0.1:40316)
0926 09:32:50.218793 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.218834 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.218905 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.219513 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.219514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.219516 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:50.220160 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:32:50.220161 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:50.220167 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:32:50.220177 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:50.220372 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:50.220373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:50.220374 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:32:50.220936 (+ 562us) client_negotiation.cc:240] Negotiation successful
0926 09:32:50.220959 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":88,"threads_started":1}
I20260926 09:32:50.221230 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 018dc56903df4c6d8849a52a1d97d6e7 (DEFAULT_TABLE table=test-table [id=60421ebfcd8a4771b59d9d98e60cf6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:50.221232 5128 tablet_service.cc:1467] Processing CreateTablet for tablet 018dc56903df4c6d8849a52a1d97d6e7 (DEFAULT_TABLE table=test-table [id=60421ebfcd8a4771b59d9d98e60cf6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:50.221585 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018dc56903df4c6d8849a52a1d97d6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:50.221585 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018dc56903df4c6d8849a52a1d97d6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:50.221854 5390 tablet_service.cc:1467] Processing CreateTablet for tablet 018dc56903df4c6d8849a52a1d97d6e7 (DEFAULT_TABLE table=test-table [id=60421ebfcd8a4771b59d9d98e60cf6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:50.222213 5390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018dc56903df4c6d8849a52a1d97d6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:50.224776 5470 tablet_bootstrap.cc:492] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: Bootstrap starting.
I20260926 09:32:50.225122 5471 tablet_bootstrap.cc:492] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Bootstrap starting.
I20260926 09:32:50.225468 5470 tablet_bootstrap.cc:654] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:50.225718 5471 tablet_bootstrap.cc:654] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:50.225798 5470 log.cc:824] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:50.226028 5471 log.cc:824] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:50.226362 5472 tablet_bootstrap.cc:492] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b: Bootstrap starting.
I20260926 09:32:50.226611 5470 tablet_bootstrap.cc:492] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: No bootstrap required, opened a new log
I20260926 09:32:50.226694 5470 ts_tablet_manager.cc:1397] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:50.226931 5472 tablet_bootstrap.cc:654] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:50.227236 5472 log.cc:824] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:50.227900 5471 tablet_bootstrap.cc:492] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: No bootstrap required, opened a new log
I20260926 09:32:50.227981 5471 ts_tablet_manager.cc:1397] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:32:50.228060 5472 tablet_bootstrap.cc:492] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b: No bootstrap required, opened a new log
I20260926 09:32:50.228128 5472 ts_tablet_manager.cc:1397] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:32:50.229360 5470 raft_consensus.cc:348] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.229625 5470 raft_consensus.cc:374] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:50.229666 5470 raft_consensus.cc:729] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef07a3cd350344bb84f34e7b057bcaea, State: Initialized, Role: FOLLOWER
I20260926 09:32:50.229984 5470 consensus_queue.cc:260] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.229980 5471 raft_consensus.cc:348] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.230161 5471 raft_consensus.cc:374] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:50.230203 5471 raft_consensus.cc:729] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc416986f304fd981fa9c53c22ee4e3, State: Initialized, Role: FOLLOWER
I20260926 09:32:50.230259 5470 ts_tablet_manager.cc:1428] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 09:32:50.230324 5471 consensus_queue.cc:260] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.230415 5456 heartbeater.cc:503] Master 127.0.98.62:41873 was elected leader, sending a full tablet report...
I20260926 09:32:50.230569 5471 ts_tablet_manager.cc:1428] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:32:50.230608 5325 heartbeater.cc:503] Master 127.0.98.62:41873 was elected leader, sending a full tablet report...
I20260926 09:32:50.231007 5472 raft_consensus.cc:348] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.231227 5472 raft_consensus.cc:374] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:50.231280 5472 raft_consensus.cc:729] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7ce94c0005475d97c4b467ae992e5b, State: Initialized, Role: FOLLOWER
I20260926 09:32:50.231431 5472 consensus_queue.cc:260] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.231698 5472 ts_tablet_manager.cc:1428] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 09:32:50.231760 5194 heartbeater.cc:503] Master 127.0.98.62:41873 was elected leader, sending a full tablet report...
W20260926 09:32:50.337265 5326 tablet.cc:2369] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:32:50.373052 5477 raft_consensus.cc:482] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:50.373184 5477 raft_consensus.cc:504] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.373615 5477 leader_election.cc:288] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f7ce94c0005475d97c4b467ae992e5b (127.0.98.1:44967), ef07a3cd350344bb84f34e7b057bcaea (127.0.98.3:34889)
I20260926 09:32:50.375022 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.373911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48705 (local address 127.0.98.1:44967)
0926 09:32:50.373965 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.373967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.374008 (+ 41us) server_negotiation.cc:407] Connection header received
0926 09:32:50.374039 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.374040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.374047 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.374101 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.374766 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.374767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.374767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.374772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.374788 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.374793 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.374880 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.374940 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.374954 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:32:50.375022 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.373838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44967 (local address 127.0.98.2:48705)
0926 09:32:50.373895 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.373908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.373954 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.374152 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.374153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.374156 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:50.374664 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:32:50.374664 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:50.374671 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:32:50.374682 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:50.374879 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:50.374880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:50.374881 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:32:50.374937 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:32:50.374953 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:32:50.375262 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.374046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48179 (local address 127.0.98.3:34889)
0926 09:32:50.374113 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.374116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.374153 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:32:50.374223 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.374224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.374230 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.374290 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.374949 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.374951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.374951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.374956 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.374969 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.374973 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.375082 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.375175 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.375193 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:32:50.375264 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.373909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34889 (local address 127.0.98.2:48179)
0926 09:32:50.374100 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.374111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.374161 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.374346 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.374347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.374360 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:50.374833 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:32:50.374833 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:50.374845 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:32:50.374861 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:50.375078 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:50.375079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:50.375079 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 09:32:50.375162 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:32:50.375177 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":106,"threads_started":1}
I20260926 09:32:50.375435 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018dc56903df4c6d8849a52a1d97d6e7" candidate_uuid: "0cc416986f304fd981fa9c53c22ee4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f7ce94c0005475d97c4b467ae992e5b" is_pre_election: true
I20260926 09:32:50.375599 5148 raft_consensus.cc:2454] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cc416986f304fd981fa9c53c22ee4e3 in term 0.
I20260926 09:32:50.375734 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018dc56903df4c6d8849a52a1d97d6e7" candidate_uuid: "0cc416986f304fd981fa9c53c22ee4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef07a3cd350344bb84f34e7b057bcaea" is_pre_election: true
I20260926 09:32:50.375880 5410 raft_consensus.cc:2454] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cc416986f304fd981fa9c53c22ee4e3 in term 0.
I20260926 09:32:50.375870 5214 leader_election.cc:302] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc416986f304fd981fa9c53c22ee4e3, 1f7ce94c0005475d97c4b467ae992e5b; no voters:
I20260926 09:32:50.375985 5477 raft_consensus.cc:2781] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:50.376067 5477 raft_consensus.cc:482] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:50.376093 5477 raft_consensus.cc:3037] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:50.376812 5477 raft_consensus.cc:504] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.376962 5477 leader_election.cc:288] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [CANDIDATE]: Term 1 election: Requested vote from peers 1f7ce94c0005475d97c4b467ae992e5b (127.0.98.1:44967), ef07a3cd350344bb84f34e7b057bcaea (127.0.98.3:34889)
I20260926 09:32:50.377113 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018dc56903df4c6d8849a52a1d97d6e7" candidate_uuid: "0cc416986f304fd981fa9c53c22ee4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f7ce94c0005475d97c4b467ae992e5b"
I20260926 09:32:50.377127 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018dc56903df4c6d8849a52a1d97d6e7" candidate_uuid: "0cc416986f304fd981fa9c53c22ee4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef07a3cd350344bb84f34e7b057bcaea"
I20260926 09:32:50.377194 5410 raft_consensus.cc:3037] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:50.377197 5148 raft_consensus.cc:3037] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:50.378057 5148 raft_consensus.cc:2454] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cc416986f304fd981fa9c53c22ee4e3 in term 1.
I20260926 09:32:50.378057 5410 raft_consensus.cc:2454] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cc416986f304fd981fa9c53c22ee4e3 in term 1.
I20260926 09:32:50.378224 5214 leader_election.cc:302] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc416986f304fd981fa9c53c22ee4e3, 1f7ce94c0005475d97c4b467ae992e5b; no voters:
I20260926 09:32:50.378336 5477 raft_consensus.cc:2781] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:50.378496 5477 raft_consensus.cc:686] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 0cc416986f304fd981fa9c53c22ee4e3, State: Running, Role: LEADER
I20260926 09:32:50.378620 5477 consensus_queue.cc:237] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } }
I20260926 09:32:50.379493 5006 catalog_manager.cc:5495] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cc416986f304fd981fa9c53c22ee4e3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0cc416986f304fd981fa9c53c22ee4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cc416986f304fd981fa9c53c22ee4e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } health_report { overall_health: HEALTHY } } }
I20260926 09:32:50.402230 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.401095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.98.2:37845)
0926 09:32:50.401172 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.401175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.401189 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:32:50.401235 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.401236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.401252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.401329 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.401993 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.401994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.401995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.401999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.402016 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.402020 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.402079 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.402134 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.402151 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:32:50.404130 5410 raft_consensus.cc:1261] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 1 FOLLOWER]: Refusing update from remote peer 0cc416986f304fd981fa9c53c22ee4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:50.404130 5148 raft_consensus.cc:1261] T 018dc56903df4c6d8849a52a1d97d6e7 P 1f7ce94c0005475d97c4b467ae992e5b [term 1 FOLLOWER]: Refusing update from remote peer 0cc416986f304fd981fa9c53c22ee4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:50.404477 5480 consensus_queue.cc:1035] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef07a3cd350344bb84f34e7b057bcaea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:50.404613 5477 consensus_queue.cc:1035] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f7ce94c0005475d97c4b467ae992e5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:50.407900 5485 mvcc.cc:205] Tried to move back new op lower bound from 7333540537972498432 to 7333540537871507456. Current Snapshot: MvccSnapshot[applied={T|T < 7333540537972498432}]
I20260926 09:32:50.409546 5006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:50.409634 5006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:50.410549 5006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57c89ee98f434eacb15ba3c4c156fe2c: Sending DeleteTablet for 3 replicas of tablet 018dc56903df4c6d8849a52a1d97d6e7
I20260926 09:32:50.410872 5259 tablet_service.cc:1514] Processing DeleteTablet for tablet 018dc56903df4c6d8849a52a1d97d6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:50 UTC) from {username='slave'} at 127.0.0.1:35336
I20260926 09:32:50.410950 5128 tablet_service.cc:1514] Processing DeleteTablet for tablet 018dc56903df4c6d8849a52a1d97d6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:50 UTC) from {username='slave'} at 127.0.0.1:53624
I20260926 09:32:50.411105 5492 tablet_replica.cc:330] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: stopping tablet replica
I20260926 09:32:50.411204 5492 raft_consensus.cc:2229] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:50.411242 5390 tablet_service.cc:1514] Processing DeleteTablet for tablet 018dc56903df4c6d8849a52a1d97d6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:50 UTC) from {username='slave'} at 127.0.0.1:40316
I20260926 09:32:50.411302 5492 raft_consensus.cc:2258] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:50.411406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 5066
I20260926 09:32:50.412551 5492 ts_tablet_manager.cc:1905] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:50.413997 5494 tablet_replica.cc:330] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: stopping tablet replica
I20260926 09:32:50.414096 5494 raft_consensus.cc:2229] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:50.414182 5494 raft_consensus.cc:2258] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:50.414511 5492 ts_tablet_manager.cc:1918] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:50.414599 5492 log.cc:1196] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/018dc56903df4c6d8849a52a1d97d6e7
I20260926 09:32:50.414623 5494 ts_tablet_manager.cc:1905] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:50.414880 5492 ts_tablet_manager.cc:1939] T 018dc56903df4c6d8849a52a1d97d6e7 P 0cc416986f304fd981fa9c53c22ee4e3: Deleting consensus metadata
I20260926 09:32:50.415271 4993 catalog_manager.cc:4841] TS 0cc416986f304fd981fa9c53c22ee4e3 (127.0.98.2:37845): tablet 018dc56903df4c6d8849a52a1d97d6e7 (table test-table [id=60421ebfcd8a4771b59d9d98e60cf6d6]) successfully deleted
I20260926 09:32:50.416391 5494 ts_tablet_manager.cc:1918] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:50.416453 5494 log.cc:1196] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/018dc56903df4c6d8849a52a1d97d6e7
I20260926 09:32:50.416640 5494 ts_tablet_manager.cc:1939] T 018dc56903df4c6d8849a52a1d97d6e7 P ef07a3cd350344bb84f34e7b057bcaea: Deleting consensus metadata
I20260926 09:32:50.416988 4990 catalog_manager.cc:4841] TS ef07a3cd350344bb84f34e7b057bcaea (127.0.98.3:34889): tablet 018dc56903df4c6d8849a52a1d97d6e7 (table test-table [id=60421ebfcd8a4771b59d9d98e60cf6d6]) successfully deleted
W20260926 09:32:50.418290 4992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44967 (error 108)
I20260926 09:32:50.418951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 5197
I20260926 09:32:50.419030 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.418896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44967 (local address 127.0.0.1:53630)
0926 09:32:50.418945 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.418993 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 09:32:50.423472 4992 catalog_manager.cc:4584] TS 1f7ce94c0005475d97c4b467ae992e5b (127.0.98.1:44967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 018dc56903df4c6d8849a52a1d97d6e7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44967: connect: Connection refused (error 111)
I20260926 09:32:50.424108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 5328
I20260926 09:32:50.428642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 4975
2026-09-26T09:32:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (735 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 09:32:50.448024 392 mini_oidc.cc:152] Starting JWKS server
I20260926 09:32:50.449162 392 webserver.cc:460] Webserver started at https://127.0.0.1:45679/ using document root <none> and password file <none>
I20260926 09:32:50.449301 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:32:50.449509 392 webserver.cc:460] Webserver started at http://127.0.0.1:44501/ using document root <none> and password file <none>
2026-09-26T09:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:50Z Disabled control of system clock
I20260926 09:32:50.457794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34333
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34333
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45679/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/testchainca.cert with env {}
W20260926 09:32:50.539629 5501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:50.539836 5501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:50.539860 5501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:50.541254 5501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:50.541309 5501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:50.541322 5501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:50.541337 5501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:50.541348 5501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:32:50.541361 5501 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:32:50.541373 5501 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45679/jwks.json
I20260926 09:32:50.543071 5501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45679/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:50.543280 5501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:50.543489 5501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:50.548717 5507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:50.548753 5509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:50.548739 5506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:50.549008 5501 server_base.cc:1034] running on GCE node
I20260926 09:32:50.549275 5501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:50.549553 5501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:50.550719 5501 hybrid_clock.cc:648] HybridClock initialized: now 1790415170550694 us; error 43 us; skew 500 ppm
I20260926 09:32:50.552225 5501 webserver.cc:460] Webserver started at http://127.0.98.62:43359/ using document root <none> and password file <none>
I20260926 09:32:50.552508 5501 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:50.552556 5501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:50.552686 5501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:50.553771 5501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "16e799b2def1375c089132567b321a19"
server_key_iv: "6fe8dfa0a19ac1607a0a338f6449eb22"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.554167 5501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "16e799b2def1375c089132567b321a19"
server_key_iv: "6fe8dfa0a19ac1607a0a338f6449eb22"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.555673 5501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:50.556504 5515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:50.556677 5501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:50.556761 5501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "16e799b2def1375c089132567b321a19"
server_key_iv: "6fe8dfa0a19ac1607a0a338f6449eb22"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.556816 5501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:50.584302 5501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:50.584610 5501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:50.584719 5501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:50.588593 5501 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34333
I20260926 09:32:50.588654 5567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34333 every 8 connection(s)
I20260926 09:32:50.588997 5501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:50.590080 5568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:50.592458 5568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f: Bootstrap starting.
I20260926 09:32:50.593065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 5501
I20260926 09:32:50.593168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:50.593183 5568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:50.593371 392 external_mini_cluster.cc:1428] Setting key 3ccdb398f4db1d7622bb187c51183033
I20260926 09:32:50.593535 5568 log.cc:824] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:50.594280 5568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f: No bootstrap required, opened a new log
I20260926 09:32:50.596345 5568 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } }
I20260926 09:32:50.596518 5568 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:50.596565 5568 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fbd28b39f94f23b2e0ac214ded0e0f, State: Initialized, Role: FOLLOWER
I20260926 09:32:50.596712 5568 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } }
I20260926 09:32:50.596807 5568 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:50.596853 5568 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:50.596900 5568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:50.597599 5568 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } }
I20260926 09:32:50.597711 5568 leader_election.cc:302] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8fbd28b39f94f23b2e0ac214ded0e0f; no voters:
I20260926 09:32:50.597891 5568 leader_election.cc:288] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:50.598014 5573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:50.598177 5568 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:50.598268 5573 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [term 1 LEADER]: Becoming Leader. State: Replica: a8fbd28b39f94f23b2e0ac214ded0e0f, State: Running, Role: LEADER
I20260926 09:32:50.598404 5573 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } }
I20260926 09:32:50.599052 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8fbd28b39f94f23b2e0ac214ded0e0f. Latest consensus state: current_term: 1 leader_uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } } }
I20260926 09:32:50.599146 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:50.599514 5574 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fbd28b39f94f23b2e0ac214ded0e0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34333 } } }
I20260926 09:32:50.599730 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.593902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.0.98.62:34333)
0926 09:32:50.594134 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.594143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.594175 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:32:50.594672 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.594679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.594741 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.594895 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.595095 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.596053 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.597039 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.597669 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.597810 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:50.598602 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.598606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.598606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.598854 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.598898 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.598915 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.598989 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.599400 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.599488 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
I20260926 09:32:50.599588 5574 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f [sys.catalog]: This master's current role is: LEADER
W20260926 09:32:50.600044 5589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:32:50.600098 5589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:32:50.600150 5583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:50.600306 5583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:50.600661 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:32:50.601903 5583 catalog_manager.cc:1347] Generated new cluster ID: 099ff34094704f7c932cda06a8790355
I20260926 09:32:50.601955 5583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:50.629863 5583 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:50.630468 5583 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:50.638715 5583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8fbd28b39f94f23b2e0ac214ded0e0f: Generated new TSK 0
I20260926 09:32:50.638950 5583 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 09:32:50.672799 5598 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 09:32:50.672920 5598 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 09:32:50.673301 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.650790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.98.62:34333)
0926 09:32:50.650877 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.650880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.651059 (+ 179us) server_negotiation.cc:407] Connection header received
0926 09:32:50.651127 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.651128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.651138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.651204 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:32:50.651426 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.651959 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.653127 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.653325 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.653400 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:50.653645 (+ 245us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:32:50.672774 (+ 19129us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 09:32:50.673209 (+ 435us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 09:32:50.673403 5572 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 09:32:50.673635 5597 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:32:50.650722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34333 (local address 127.0.0.1:42166)
0926 09:32:50.650942 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.650955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.651032 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.651246 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.651247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.651250 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:32:50.651347 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:50.651350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.652066 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.652067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:50.653025 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:50.653028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.653429 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.653433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:50.653500 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:50.653524 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:32:50.673163 (+ 19639us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 09:32:50.673289 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34333: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":127,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34333: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 09:32:50.674252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 5501
2026-09-26T09:32:50Z chronyd exiting
I20260926 09:32:50.710317 392 test_util.cc:175] -----------------------------------------------
I20260926 09:32:50.710444 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790415146110679-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (263 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 09:32:50.711272 392 mini_oidc.cc:152] Starting JWKS server
I20260926 09:32:50.712059 392 webserver.cc:460] Webserver started at https://127.0.0.1:34081/ using document root <none> and password file <none>
I20260926 09:32:50.712102 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:32:50.712285 392 webserver.cc:460] Webserver started at http://127.0.0.1:46077/ using document root <none> and password file <none>
2026-09-26T09:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:50Z Disabled control of system clock
I20260926 09:32:50.720274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43633
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43633
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34081/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/testchainca.cert with env {}
W20260926 09:32:50.805881 5604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:50.806165 5604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:50.806195 5604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:50.807614 5604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:50.807677 5604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:50.807694 5604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:50.807711 5604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:50.807726 5604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:32:50.807744 5604 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:32:50.807758 5604 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34081/jwks.json
I20260926 09:32:50.809513 5604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34081/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:50.809765 5604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:50.809991 5604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:50.815531 5612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:50.815570 5610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:50.815653 5604 server_base.cc:1034] running on GCE node
W20260926 09:32:50.815573 5609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:50.815976 5604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:50.816330 5604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:50.817492 5604 hybrid_clock.cc:648] HybridClock initialized: now 1790415170817467 us; error 41 us; skew 500 ppm
I20260926 09:32:50.818753 5604 webserver.cc:460] Webserver started at http://127.0.98.62:44897/ using document root <none> and password file <none>
I20260926 09:32:50.818961 5604 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:50.819005 5604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:50.819128 5604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:50.820319 5604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "45c3e9838b724920a995ebd9a94987ff"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "4dd83d1c48d9f9d3793ba16c20065403"
server_key_iv: "6e9b8861afc9ccc9301b79a5196a2b90"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.820642 5604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "45c3e9838b724920a995ebd9a94987ff"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "4dd83d1c48d9f9d3793ba16c20065403"
server_key_iv: "6e9b8861afc9ccc9301b79a5196a2b90"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.821945 5604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:32:50.822695 5618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:50.822893 5604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:50.823000 5604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "45c3e9838b724920a995ebd9a94987ff"
format_stamp: "Formatted at 2026-09-26 09:32:50 on dist-test-slave-21h8"
server_key: "4dd83d1c48d9f9d3793ba16c20065403"
server_key_iv: "6e9b8861afc9ccc9301b79a5196a2b90"
server_key_version: "encryptionkey@0"
I20260926 09:32:50.823110 5604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:50.835860 5604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:50.836227 5604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:50.836393 5604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:50.840929 5604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43633
I20260926 09:32:50.840977 5670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43633 every 8 connection(s)
I20260926 09:32:50.841367 5604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:50.842358 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:50.844808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 5604
I20260926 09:32:50.844828 5671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff: Bootstrap starting.
I20260926 09:32:50.844924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:50.845115 392 external_mini_cluster.cc:1428] Setting key 67f2173662f3d3f953118b460a2c7e29
I20260926 09:32:50.845588 5671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:50.845934 5671 log.cc:824] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:50.846735 5671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff: No bootstrap required, opened a new log
I20260926 09:32:50.849141 5671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3e9838b724920a995ebd9a94987ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } }
I20260926 09:32:50.849325 5671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:50.849356 5671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c3e9838b724920a995ebd9a94987ff, State: Initialized, Role: FOLLOWER
I20260926 09:32:50.849478 5671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3e9838b724920a995ebd9a94987ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } }
I20260926 09:32:50.849550 5671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:50.849579 5671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:50.849618 5671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:50.850436 5671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3e9838b724920a995ebd9a94987ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } }
I20260926 09:32:50.850576 5671 leader_election.cc:302] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45c3e9838b724920a995ebd9a94987ff; no voters:
I20260926 09:32:50.850785 5671 leader_election.cc:288] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:50.850847 5676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:50.850973 5676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [term 1 LEADER]: Becoming Leader. State: Replica: 45c3e9838b724920a995ebd9a94987ff, State: Running, Role: LEADER
I20260926 09:32:50.851054 5676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3e9838b724920a995ebd9a94987ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } }
I20260926 09:32:50.851076 5671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:50.851423 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45c3e9838b724920a995ebd9a94987ff. Latest consensus state: current_term: 1 leader_uuid: "45c3e9838b724920a995ebd9a94987ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3e9838b724920a995ebd9a94987ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } } }
I20260926 09:32:50.851502 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:50.851439 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45c3e9838b724920a995ebd9a94987ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3e9838b724920a995ebd9a94987ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } } }
I20260926 09:32:50.851693 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:50.851758 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.845678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.98.62:43633)
0926 09:32:50.845902 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.845910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.845941 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:32:50.846447 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.846455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.846519 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.846650 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:50.846930 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.848359 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.849160 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.849789 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.849909 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:50.850709 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:50.850713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:50.850714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:50.850945 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:50.850995 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:50.851012 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:50.851367 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:50.851401 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 09:32:50.851493 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260926 09:32:50.851781 5682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:50.852330 5682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:50.852891 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:32:50.854065 5682 catalog_manager.cc:1347] Generated new cluster ID: 7cfa98763e5a4fd2aeafe42b141c6e09
I20260926 09:32:50.854117 5682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:50.865347 5682 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:50.865911 5682 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:50.869830 5682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45c3e9838b724920a995ebd9a94987ff: Generated new TSK 0
I20260926 09:32:50.870047 5682 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 09:32:50.890588 5701 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 09:32:50.890712 5701 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 09:32:50.891021 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:50.875456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.0.98.62:43633)
0926 09:32:50.875538 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:50.875540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:50.875643 (+ 103us) server_negotiation.cc:407] Connection header received
0926 09:32:50.875719 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:50.875720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:50.875735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:50.875823 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:32:50.875985 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.876779 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.877607 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.877929 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.878003 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:50.878151 (+ 148us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:32:50.890773 (+ 12622us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 09:32:50.890943 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 09:32:50.891098 5674 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 09:32:50.891135 5700 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:32:50.875414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43633 (local address 127.0.0.1:33982)
0926 09:32:50.875564 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:50.875576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:50.875644 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:50.875852 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:50.875853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:50.875855 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:32:50.875916 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:50.875918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.876902 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.876903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:50.877512 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:50.877515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:50.878008 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:50.878013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:50.878054 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:50.878060 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:32:50.890921 (+ 12861us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 09:32:50.891021 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43633: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260926 09:32:50.891479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 5604
2026-09-26T09:32:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 09:32:50.929491 392 mini_oidc.cc:152] Starting JWKS server
I20260926 09:32:50.930218 392 webserver.cc:460] Webserver started at https://127.0.0.1:42397/ using document root <none> and password file <none>
I20260926 09:32:50.930263 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:32:50.930429 392 webserver.cc:460] Webserver started at http://127.0.0.1:37797/ using document root <none> and password file <none>
2026-09-26T09:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:50Z Disabled control of system clock
I20260926 09:32:50.938297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39403
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39403
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 09:32:51.022281 5707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:51.022548 5707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:51.022588 5707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:51.023919 5707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:51.023998 5707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:51.024020 5707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:51.024041 5707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:51.024070 5707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:32:51.024096 5707 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:32:51.024117 5707 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42397/jwks.json
I20260926 09:32:51.025555 5707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:51.025864 5707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:51.026183 5707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:51.031396 5715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:51.031405 5713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:51.031383 5712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:51.031698 5707 server_base.cc:1034] running on GCE node
I20260926 09:32:51.032027 5707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:51.032400 5707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:51.033584 5707 hybrid_clock.cc:648] HybridClock initialized: now 1790415171033565 us; error 51 us; skew 500 ppm
I20260926 09:32:51.034930 5707 webserver.cc:460] Webserver started at http://127.0.98.62:44109/ using document root <none> and password file <none>
I20260926 09:32:51.035223 5707 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:51.035302 5707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:51.035490 5707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:51.036657 5707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "587732b997ce4fe69fa5af1904297f7f"
format_stamp: "Formatted at 2026-09-26 09:32:51 on dist-test-slave-21h8"
server_key: "d0b7c40ee5d9196e637aedd400d2a529"
server_key_iv: "69d59532a2e9ffd66c2e086516854ef3"
server_key_version: "encryptionkey@0"
I20260926 09:32:51.037055 5707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "587732b997ce4fe69fa5af1904297f7f"
format_stamp: "Formatted at 2026-09-26 09:32:51 on dist-test-slave-21h8"
server_key: "d0b7c40ee5d9196e637aedd400d2a529"
server_key_iv: "69d59532a2e9ffd66c2e086516854ef3"
server_key_version: "encryptionkey@0"
I20260926 09:32:51.038486 5707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:32:51.039373 5721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:51.039585 5707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:51.039695 5707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "587732b997ce4fe69fa5af1904297f7f"
format_stamp: "Formatted at 2026-09-26 09:32:51 on dist-test-slave-21h8"
server_key: "d0b7c40ee5d9196e637aedd400d2a529"
server_key_iv: "69d59532a2e9ffd66c2e086516854ef3"
server_key_version: "encryptionkey@0"
I20260926 09:32:51.039783 5707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:51.070190 5707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:51.070576 5707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:51.070745 5707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:51.074909 5707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39403
I20260926 09:32:51.074949 5773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39403 every 8 connection(s)
I20260926 09:32:51.075336 5707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:51.076081 5774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:51.078984 5774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f: Bootstrap starting.
I20260926 09:32:51.079716 5774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:51.080036 5774 log.cc:824] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:51.080803 5774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f: No bootstrap required, opened a new log
I20260926 09:32:51.082427 5774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587732b997ce4fe69fa5af1904297f7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39403 } }
I20260926 09:32:51.082587 5774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:51.082612 5774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587732b997ce4fe69fa5af1904297f7f, State: Initialized, Role: FOLLOWER
I20260926 09:32:51.082779 5774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587732b997ce4fe69fa5af1904297f7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39403 } }
I20260926 09:32:51.082842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 5707
I20260926 09:32:51.082894 5774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:51.082935 5774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:51.082960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:51.082986 5774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:51.083155 392 external_mini_cluster.cc:1428] Setting key fa9dee24cff333444950c7fe2af88f03
I20260926 09:32:51.083847 5774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587732b997ce4fe69fa5af1904297f7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39403 } }
I20260926 09:32:51.083966 5774 leader_election.cc:302] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587732b997ce4fe69fa5af1904297f7f; no voters:
I20260926 09:32:51.084123 5774 leader_election.cc:288] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:51.084184 5779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:51.084299 5779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [term 1 LEADER]: Becoming Leader. State: Replica: 587732b997ce4fe69fa5af1904297f7f, State: Running, Role: LEADER
I20260926 09:32:51.084380 5779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587732b997ce4fe69fa5af1904297f7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39403 } }
I20260926 09:32:51.084420 5774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:51.084890 5781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "587732b997ce4fe69fa5af1904297f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587732b997ce4fe69fa5af1904297f7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39403 } } }
I20260926 09:32:51.085037 5781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:51.085381 5786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:51.085937 5786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:51.086066 5780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587732b997ce4fe69fa5af1904297f7f. Latest consensus state: current_term: 1 leader_uuid: "587732b997ce4fe69fa5af1904297f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587732b997ce4fe69fa5af1904297f7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39403 } } }
I20260926 09:32:51.086153 5780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:51.087816 5786 catalog_manager.cc:1347] Generated new cluster ID: 0382ba11a3724ed5ad094db13e9095af
I20260926 09:32:51.087872 5786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:51.089735 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:51.083677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.0.98.62:39403)
0926 09:32:51.083942 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:51.083950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:51.083999 (+ 49us) server_negotiation.cc:407] Connection header received
0926 09:32:51.084121 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:51.084127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:51.084214 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:51.084359 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:51.084731 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.086201 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.087475 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.088100 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.088221 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:51.088990 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:51.088994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:51.088995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:51.089211 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:51.089252 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:51.089266 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:51.089328 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:51.089446 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 09:32:51.089525 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
I20260926 09:32:51.090812 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:32:51.105997 5786 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:51.106618 5786 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:51.111909 5786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 587732b997ce4fe69fa5af1904297f7f: Generated new TSK 0
I20260926 09:32:51.112149 5786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:32:51.124331 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:51.120376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.98.62:39403)
0926 09:32:51.120441 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:51.120443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:51.120588 (+ 145us) server_negotiation.cc:407] Connection header received
0926 09:32:51.120633 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:51.120634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:51.120647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:51.120719 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:32:51.120837 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.121646 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.122471 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.122781 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.122855 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:51.123041 (+ 186us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:32:51.124102 (+ 1061us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 09:32:51.124243 (+ 141us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57734: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 09:32:51.124398 5778 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57734: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 09:32:51.124498 5803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:32:51.120298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39403 (local address 127.0.0.1:57734)
0926 09:32:51.120465 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:51.120476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:51.120543 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:51.120716 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:51.120716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:51.120718 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:32:51.120767 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:51.120769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.121749 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.121749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:51.122370 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:51.122372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.122869 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.122877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:51.122917 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:51.122923 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:32:51.124246 (+ 1323us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 09:32:51.124380 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39403: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260926 09:32:51.125054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 5707
E20260926 09:32:51.137461 5804 webserver.cc:533] Webserver: sslize failed:
E20260926 09:32:51.137562 5804 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T09:32:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T09:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:51Z Disabled control of system clock
I20260926 09:32:51.171161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44425
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44425
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/testchainca.cert with env {}
W20260926 09:32:51.254899 5808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:51.255132 5808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:51.255156 5808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:51.256440 5808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:51.256493 5808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:51.256506 5808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:51.256520 5808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:51.256533 5808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:32:51.256546 5808 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:32:51.256596 5808 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 09:32:51.258168 5808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:51.258456 5808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:51.258703 5808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:51.263731 5814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:51.263790 5816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:51.263792 5813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:51.264120 5808 server_base.cc:1034] running on GCE node
I20260926 09:32:51.264400 5808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:51.264773 5808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:51.265955 5808 hybrid_clock.cc:648] HybridClock initialized: now 1790415171265926 us; error 42 us; skew 500 ppm
I20260926 09:32:51.267475 5808 webserver.cc:460] Webserver started at http://127.0.98.62:34045/ using document root <none> and password file <none>
I20260926 09:32:51.267777 5808 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:51.267831 5808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:51.268029 5808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:51.269203 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "8274f945159e425a98f5131b6b456006"
format_stamp: "Formatted at 2026-09-26 09:32:51 on dist-test-slave-21h8"
server_key: "1441d4d1cf0f19b233fdd017f44be1ae"
server_key_iv: "4bddc9e03015bc9f383afdfe9706a75a"
server_key_version: "encryptionkey@0"
I20260926 09:32:51.269651 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "8274f945159e425a98f5131b6b456006"
format_stamp: "Formatted at 2026-09-26 09:32:51 on dist-test-slave-21h8"
server_key: "1441d4d1cf0f19b233fdd017f44be1ae"
server_key_iv: "4bddc9e03015bc9f383afdfe9706a75a"
server_key_version: "encryptionkey@0"
I20260926 09:32:51.271121 5808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:32:51.272053 5822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:51.272248 5808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:51.272357 5808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "8274f945159e425a98f5131b6b456006"
format_stamp: "Formatted at 2026-09-26 09:32:51 on dist-test-slave-21h8"
server_key: "1441d4d1cf0f19b233fdd017f44be1ae"
server_key_iv: "4bddc9e03015bc9f383afdfe9706a75a"
server_key_version: "encryptionkey@0"
I20260926 09:32:51.272447 5808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:51.290262 5808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:51.290668 5808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:51.290844 5808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:51.295816 5808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44425
I20260926 09:32:51.295850 5874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44425 every 8 connection(s)
I20260926 09:32:51.296267 5808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:51.297235 5875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:51.299713 5875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006: Bootstrap starting.
I20260926 09:32:51.300310 5875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:51.300611 5875 log.cc:824] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:51.301321 5875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006: No bootstrap required, opened a new log
I20260926 09:32:51.302948 5875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8274f945159e425a98f5131b6b456006" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } }
I20260926 09:32:51.303126 5875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:51.303159 5875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8274f945159e425a98f5131b6b456006, State: Initialized, Role: FOLLOWER
I20260926 09:32:51.303406 5875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8274f945159e425a98f5131b6b456006" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } }
I20260926 09:32:51.303493 5875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:51.303539 5875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:51.303599 5875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:51.304193 5875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8274f945159e425a98f5131b6b456006" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } }
I20260926 09:32:51.304291 5875 leader_election.cc:302] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8274f945159e425a98f5131b6b456006; no voters:
I20260926 09:32:51.304525 5875 leader_election.cc:288] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:51.304560 5878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:51.304818 5878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [term 1 LEADER]: Becoming Leader. State: Replica: 8274f945159e425a98f5131b6b456006, State: Running, Role: LEADER
I20260926 09:32:51.304819 5875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:51.304963 5878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8274f945159e425a98f5131b6b456006" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } }
I20260926 09:32:51.305398 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8274f945159e425a98f5131b6b456006. Latest consensus state: current_term: 1 leader_uuid: "8274f945159e425a98f5131b6b456006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8274f945159e425a98f5131b6b456006" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } } }
I20260926 09:32:51.305526 5879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8274f945159e425a98f5131b6b456006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8274f945159e425a98f5131b6b456006" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44425 } } }
I20260926 09:32:51.305595 5879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:51.305563 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:51.305989 5887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:51.306309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 5808
I20260926 09:32:51.306422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:51.306640 392 external_mini_cluster.cc:1428] Setting key 3e6bfefbe525339819d7fa3dde61cb84
I20260926 09:32:51.306640 5887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:51.308508 5887 catalog_manager.cc:1347] Generated new cluster ID: 6b56fcf825ae465dbd9efcafe6a17029
I20260926 09:32:51.308573 5887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:51.313784 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:51.307280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.0.98.62:44425)
0926 09:32:51.307581 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:51.307591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:51.307625 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:32:51.307751 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:51.307759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:51.307837 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:51.308060 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:51.308328 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.309826 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.310993 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.311736 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.311862 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:51.312683 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:51.312687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:51.312688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:32:51.312918 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:51.312971 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:32:51.312988 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:51.313055 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:51.313213 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 09:32:51.313401 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260926 09:32:51.314889 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:32:51.337713 5887 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:51.338995 5887 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:51.343580 5887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8274f945159e425a98f5131b6b456006: Generated new TSK 0
I20260926 09:32:51.343791 5887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:32:51.511499 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:51.353516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.0.98.62:44425)
0926 09:32:51.353565 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:51.353572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:51.353632 (+ 60us) server_negotiation.cc:407] Connection header received
0926 09:32:51.353728 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:51.353728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:51.353750 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:51.353815 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:32:51.353966 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.354471 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.355675 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.355885 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.355946 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:51.356179 (+ 233us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:32:51.511167 (+154988us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 09:32:51.511437 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34218: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 09:32:51.511569 5897 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34218: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 09:32:51.511790 5904 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:32:51.353359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44425 (local address 127.0.0.1:34218)
0926 09:32:51.353569 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:51.353581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:51.353651 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:51.353833 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:51.353833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:51.353836 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:32:51.353902 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:51.353904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.354568 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.354569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:51.355585 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:51.355588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:51.355970 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:51.355978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:51.356039 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:51.356046 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:32:51.511435 (+155389us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 09:32:51.511640 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":132,"threads_started":1}
I20260926 09:32:51.512199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 5808
2026-09-26T09:32:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (366 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:51 dist-test-slave-21h8 krb5kdc[5916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:51 dist-test-slave-21h8 krb5kdc[5916](info): set up 2 sockets
Sep 26 09:32:51 dist-test-slave-21h8 krb5kdc[5916](info): commencing operation
krb5kdc: starting...
W20260926 09:32:52.747661 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:52 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:32:52.870813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:52.953616 5929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:52.953821 5929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:52.953842 5929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:52.954919 5929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:52.954955 5929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:52.954969 5929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:52.954982 5929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:52.954994 5929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:52.956482 5929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:52.956692 5929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:52.956862 5929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:52.961963 5937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:52.961987 5929 server_base.cc:1034] running on GCE node
W20260926 09:32:52.961958 5934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:52.961958 5935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:52.962572 5929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:52.962903 5929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:52.964100 5929 hybrid_clock.cc:648] HybridClock initialized: now 1790415172964078 us; error 29 us; skew 500 ppm
Sep 26 09:32:52 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:52.967137 5929 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:32:52.967644 5929 webserver.cc:460] Webserver started at http://127.0.98.62:36125/ using document root <none> and password file <none>
I20260926 09:32:52.967851 5929 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:52.967892 5929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:52.967983 5929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:52.969014 5929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "fe73d4ae9e8b4026820a963774c0562f"
format_stamp: "Formatted at 2026-09-26 09:32:52 on dist-test-slave-21h8"
server_key: "0e56c1bf458d0464b84e074716ccb62d"
server_key_iv: "82b433102bcc6efc440b75638f393a07"
server_key_version: "encryptionkey@0"
I20260926 09:32:52.969313 5929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe73d4ae9e8b4026820a963774c0562f"
format_stamp: "Formatted at 2026-09-26 09:32:52 on dist-test-slave-21h8"
server_key: "0e56c1bf458d0464b84e074716ccb62d"
server_key_iv: "82b433102bcc6efc440b75638f393a07"
server_key_version: "encryptionkey@0"
I20260926 09:32:52.970515 5929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:52.971148 5944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:52.971357 5929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:52.971436 5929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "fe73d4ae9e8b4026820a963774c0562f"
format_stamp: "Formatted at 2026-09-26 09:32:52 on dist-test-slave-21h8"
server_key: "0e56c1bf458d0464b84e074716ccb62d"
server_key_iv: "82b433102bcc6efc440b75638f393a07"
server_key_version: "encryptionkey@0"
I20260926 09:32:52.971537 5929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:52.983981 5929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:52.986084 5929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:52.986238 5929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:52.990396 5929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43797
I20260926 09:32:52.990674 5996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43797 every 8 connection(s)
I20260926 09:32:52.990779 5929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:52.991595 5997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:52.994096 5997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f: Bootstrap starting.
I20260926 09:32:52.994760 5997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:52.995095 5997 log.cc:824] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:52.995577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 5929
I20260926 09:32:52.995683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:52.995913 392 external_mini_cluster.cc:1428] Setting key 247ceb956fa72e4e92642d6d3ce69c07
I20260926 09:32:52.996099 5997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f: No bootstrap required, opened a new log
I20260926 09:32:52.998045 5997 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe73d4ae9e8b4026820a963774c0562f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43797 } }
I20260926 09:32:52.998242 5997 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:52.998288 5997 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe73d4ae9e8b4026820a963774c0562f, State: Initialized, Role: FOLLOWER
I20260926 09:32:52.998445 5997 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe73d4ae9e8b4026820a963774c0562f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43797 } }
I20260926 09:32:52.999002 5997 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:52.999053 5997 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:52.999119 5997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:52.999809 5997 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe73d4ae9e8b4026820a963774c0562f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43797 } }
I20260926 09:32:52.999912 5997 leader_election.cc:302] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe73d4ae9e8b4026820a963774c0562f; no voters:
I20260926 09:32:53.000109 5997 leader_election.cc:288] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:53.000140 6002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:53.000298 6002 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [term 1 LEADER]: Becoming Leader. State: Replica: fe73d4ae9e8b4026820a963774c0562f, State: Running, Role: LEADER
I20260926 09:32:53.000377 5997 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:53.000440 6002 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe73d4ae9e8b4026820a963774c0562f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43797 } }
I20260926 09:32:53.001689 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe73d4ae9e8b4026820a963774c0562f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe73d4ae9e8b4026820a963774c0562f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43797 } } }
I20260926 09:32:53.001794 6004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe73d4ae9e8b4026820a963774c0562f. Latest consensus state: current_term: 1 leader_uuid: "fe73d4ae9e8b4026820a963774c0562f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe73d4ae9e8b4026820a963774c0562f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43797 } } }
I20260926 09:32:53.001801 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:53.001860 6004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f [sys.catalog]: This master's current role is: LEADER
W20260926 09:32:53.002485 6017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:32:53.002586 6017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:32:53.002684 6018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:53.002928 6018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:53.005355 6018 catalog_manager.cc:1347] Generated new cluster ID: 7264899f30024bcc8f562ed018e6993c
I20260926 09:32:53.005419 6018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:53.009354 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:52.996475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.0.98.62:43797)
0926 09:32:52.996660 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:52.996669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:52.996701 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:32:52.998029 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:52.998035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:52.998082 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:52.998198 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:52.999161 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.000361 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.001624 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.002387 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.002529 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.005453 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:53.005457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:53.005457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:53.005469 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:53.007650 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.008072 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.008075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.008075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.008202 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.008432 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.008433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.008433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.008736 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:53.008902 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:53.009008 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 09:32:53.009147 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260926 09:32:53.021193 6018 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:53.021955 6018 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:53.026894 6018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe73d4ae9e8b4026820a963774c0562f: Generated new TSK 0
I20260926 09:32:53.027138 6018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:32:53.058677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43797
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:53.148568 6023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:53.148777 6023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:53.148799 6023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:53.149966 6023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:53.150009 6023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:53.150036 6023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:53.151607 6023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:53.151858 6023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:53.152119 6023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:53.157310 6031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:53.157320 6023 server_base.cc:1034] running on GCE node
W20260926 09:32:53.157326 6029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:53.157320 6028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:53.157738 6023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:53.158007 6023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:53.159245 6023 hybrid_clock.cc:648] HybridClock initialized: now 1790415173159225 us; error 36 us; skew 500 ppm
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.161988 6023 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:32:53.162506 6023 webserver.cc:460] Webserver started at http://127.0.98.1:33705/ using document root <none> and password file <none>
I20260926 09:32:53.162760 6023 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:53.162809 6023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:53.163005 6023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:53.164136 6023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1dbce71bc6e4bf9885bf863107f7b40"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "a1fdfe4dae5b4f5056d01a434f23a3c8"
server_key_iv: "f5f9882bac6861fce9960cd4ec9e0b1c"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.164497 6023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1dbce71bc6e4bf9885bf863107f7b40"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "a1fdfe4dae5b4f5056d01a434f23a3c8"
server_key_iv: "f5f9882bac6861fce9960cd4ec9e0b1c"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.165874 6023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:32:53.166689 6038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.166891 6023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:53.166978 6023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "c1dbce71bc6e4bf9885bf863107f7b40"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "a1fdfe4dae5b4f5056d01a434f23a3c8"
server_key_iv: "f5f9882bac6861fce9960cd4ec9e0b1c"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.167042 6023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:53.178450 6023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:53.180517 6023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:53.180632 6023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:53.180819 6023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:53.181159 6023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:53.181193 6023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.181217 6023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:53.181269 6023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.186785 6023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36975
I20260926 09:32:53.186885 6151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36975 every 8 connection(s)
I20260926 09:32:53.187207 6023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:53.194322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 6023
I20260926 09:32:53.194480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:53.194715 392 external_mini_cluster.cc:1428] Setting key 8bd7d4678471657a7cfa3069650989e2
I20260926 09:32:53.198354 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.188060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50801 (local address 127.0.98.62:43797)
0926 09:32:53.188126 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.188128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.188363 (+ 235us) server_negotiation.cc:407] Connection header received
0926 09:32:53.189366 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.189368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.189385 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.189439 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.190127 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.190639 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.191833 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.192047 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.192139 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.194350 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:53.194352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:53.194352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:53.194361 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:53.196500 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.197073 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.197075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.197075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.197162 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.197530 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.197531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.197532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.197659 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:53.197722 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:53.198163 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 09:32:53.198278 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:32:53.198597 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.188042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43797 (local address 127.0.98.1:50801)
0926 09:32:53.188227 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:53.188269 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:53.189238 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:53.189513 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:53.189518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:53.189804 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:53.190001 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:53.190005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.190748 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.190749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:53.191721 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:53.191724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.192146 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.192147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:53.192280 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.192803 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:53.192812 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:53.194230 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:53.196609 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:53.196618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:53.196620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:53.196966 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:53.197249 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:53.197251 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:53.197251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:53.197426 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:53.197737 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:53.197742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:53.197935 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 09:32:53.198082 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 09:32:53.198285 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260926 09:32:53.199235 6152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43797
I20260926 09:32:53.199391 6152 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:53.199663 6152 heartbeater.cc:511] Master 127.0.98.62:43797 requested a full tablet report, sending...
I20260926 09:32:53.200313 5961 ts_manager.cc:194] Registered new tserver with Master: c1dbce71bc6e4bf9885bf863107f7b40 (127.0.98.1:36975)
I20260926 09:32:53.201097 5961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50801
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:32:53.223374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43797
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:53.306910 6157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:53.307202 6157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:53.307245 6157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:53.308504 6157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:53.308596 6157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:53.308646 6157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:53.310266 6157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:53.310573 6157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:53.310844 6157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:53.316401 6165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:53.316409 6162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:53.316442 6163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:53.316426 6157 server_base.cc:1034] running on GCE node
I20260926 09:32:53.316998 6157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:53.317291 6157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:53.318434 6157 hybrid_clock.cc:648] HybridClock initialized: now 1790415173318416 us; error 38 us; skew 500 ppm
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.321197 6157 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:32:53.321698 6157 webserver.cc:460] Webserver started at http://127.0.98.2:44985/ using document root <none> and password file <none>
I20260926 09:32:53.321949 6157 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:53.322032 6157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:53.322197 6157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:53.323371 6157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "2dea0e52741449abad03e9beb99daf88"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "66bdd0e74db32a7e91a9580f99e40a23"
server_key_iv: "ab14bc3b7bb9307d3e31d12e8a14bbb2"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.323799 6157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2dea0e52741449abad03e9beb99daf88"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "66bdd0e74db32a7e91a9580f99e40a23"
server_key_iv: "ab14bc3b7bb9307d3e31d12e8a14bbb2"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.325263 6157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:53.326102 6172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.326274 6157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:53.326383 6157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "2dea0e52741449abad03e9beb99daf88"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "66bdd0e74db32a7e91a9580f99e40a23"
server_key_iv: "ab14bc3b7bb9307d3e31d12e8a14bbb2"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.326476 6157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:53.338869 6157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:53.341389 6157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:53.341593 6157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:53.341900 6157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:53.342285 6157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:53.342334 6157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.342377 6157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:53.342399 6157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.348865 6157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33729
I20260926 09:32:53.348903 6285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33729 every 8 connection(s)
I20260926 09:32:53.349256 6157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:53.358423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 6157
I20260926 09:32:53.358531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:53.358742 392 external_mini_cluster.cc:1428] Setting key 4c97facd67990054bb837225b3ce2009
I20260926 09:32:53.359475 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.350154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44087 (local address 127.0.98.62:43797)
0926 09:32:53.350200 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.350202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.350445 (+ 243us) server_negotiation.cc:407] Connection header received
0926 09:32:53.351492 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.351493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.351505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.351585 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.352280 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.352788 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.353824 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.354026 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.354106 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.356260 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:53.356261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:53.356262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:53.356269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:53.357751 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.358184 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.358219 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.358219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.358301 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.358629 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.358630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.358631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.358759 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:53.358853 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:53.359268 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 09:32:53.359373 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:53.359771 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.350114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43797 (local address 127.0.98.2:44087)
0926 09:32:53.350318 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:53.350354 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:53.351375 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:53.351700 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:53.351704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:53.351963 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:53.352165 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:53.352170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.352896 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.352898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:53.353709 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:53.353711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.354124 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.354125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:53.354250 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.354746 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:53.354755 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:53.356162 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:53.357833 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:53.357839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:53.357841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:53.358074 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:53.358387 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:53.358388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:53.358388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:53.358536 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:53.358900 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:53.358906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:53.359101 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 09:32:53.359219 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 09:32:53.359460 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260926 09:32:53.360373 6286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43797
I20260926 09:32:53.360476 6286 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:53.360702 6286 heartbeater.cc:511] Master 127.0.98.62:43797 requested a full tablet report, sending...
I20260926 09:32:53.361124 5961 ts_manager.cc:194] Registered new tserver with Master: 2dea0e52741449abad03e9beb99daf88 (127.0.98.2:33729)
I20260926 09:32:53.361529 5961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44087
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:32:53.388025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43797
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:53.471021 6291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:53.471422 6291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:53.471455 6291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:53.472718 6291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:53.472762 6291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:53.472790 6291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:53.474347 6291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:53.474817 6291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:53.475101 6291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:32:53.480278 6291 server_base.cc:1034] running on GCE node
W20260926 09:32:53.480208 6299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:53.480365 6297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:53.480208 6296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:53.480754 6291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:53.481184 6291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:53.482381 6291 hybrid_clock.cc:648] HybridClock initialized: now 1790415173482374 us; error 53 us; skew 500 ppm
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.485020 6291 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:32:53.485425 6291 webserver.cc:460] Webserver started at http://127.0.98.3:42009/ using document root <none> and password file <none>
I20260926 09:32:53.485616 6291 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:53.485656 6291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:53.485749 6291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:53.486773 6291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "10e16fcc8efd4d37a610898acfc6d1ac"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "51cc6863b5c951f038a1182cdc9f86c9"
server_key_iv: "4614d49a62978d8fccf4af73bc866c1b"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.487073 6291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "10e16fcc8efd4d37a610898acfc6d1ac"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "51cc6863b5c951f038a1182cdc9f86c9"
server_key_iv: "4614d49a62978d8fccf4af73bc866c1b"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.488546 6291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:32:53.489331 6306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.489517 6291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:53.489602 6291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "10e16fcc8efd4d37a610898acfc6d1ac"
format_stamp: "Formatted at 2026-09-26 09:32:53 on dist-test-slave-21h8"
server_key: "51cc6863b5c951f038a1182cdc9f86c9"
server_key_iv: "4614d49a62978d8fccf4af73bc866c1b"
server_key_version: "encryptionkey@0"
I20260926 09:32:53.489660 6291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:53.505936 6291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:53.508088 6291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:53.508205 6291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:53.508402 6291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:53.508721 6291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:53.508759 6291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.508785 6291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:53.508828 6291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:53.514918 6291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44237
I20260926 09:32:53.514958 6419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44237 every 8 connection(s)
I20260926 09:32:53.515305 6291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:32:53.522791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 6291
I20260926 09:32:53.522859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:53.523027 392 external_mini_cluster.cc:1428] Setting key 7be642499fe37bda128b3206f6b5ace3
I20260926 09:32:53.529822 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.516151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51107 (local address 127.0.98.62:43797)
0926 09:32:53.516241 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.516243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.516415 (+ 172us) server_negotiation.cc:407] Connection header received
0926 09:32:53.517465 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.517466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.517477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.517554 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.518190 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.518683 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.519912 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.520102 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.520159 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.522406 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:53.522408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:53.522408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:53.522416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:53.528426 (+ 6010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.528878 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.528878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.528879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.528936 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.529191 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.529192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.529192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.529300 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:53.529406 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:53.529689 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 09:32:53.529755 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 09:32:53.530092 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.516125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43797 (local address 127.0.98.3:51107)
0926 09:32:53.516281 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:53.516325 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:53.517361 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:53.517642 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:53.517646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:53.517908 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:53.518113 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:53.518118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.518795 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.518796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:53.519834 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:53.519836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.520200 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.520201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:53.520311 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.520866 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:53.520874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:53.522296 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:53.528518 (+ 6222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:53.528527 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:53.528530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:53.528801 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:53.529016 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:53.529017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:53.529017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:53.529129 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:53.529415 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:53.529421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:53.529568 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 09:32:53.529658 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:32:53.529797 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260926 09:32:53.530665 6420 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43797
I20260926 09:32:53.530781 6420 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:53.530967 6420 heartbeater.cc:511] Master 127.0.98.62:43797 requested a full tablet report, sending...
I20260926 09:32:53.531417 5961 ts_manager.cc:194] Registered new tserver with Master: 10e16fcc8efd4d37a610898acfc6d1ac (127.0.98.3:44237)
I20260926 09:32:53.531813 5961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51107
I20260926 09:32:53.534520 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/test-admin.keytab.
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.552290 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 09:32:53.554028 6427 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:32:53.553222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.0.98.1:36975)
0926 09:32:53.553392 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.553397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.553406 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:53.553430 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.553432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.553445 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.553534 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.553943 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51422: BlockingRecv error: recv got EOF from 127.0.0.1:51422 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
W20260926 09:32:53.554133 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.555160 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.556154 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:53.564731 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.556687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.0.98.1:36975)
0926 09:32:53.556746 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.556748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.556759 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:32:53.556804 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.556805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.556815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.556868 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.557255 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.557957 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.559093 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.559508 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.559571 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.561650 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:53.561659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:53.561660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:53.561667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:53.563265 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.563820 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.563824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.563824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.563891 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.564140 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.564141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.564142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.564294 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:53.564481 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:53.564530 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 09:32:53.564610 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 09:32:53.566871 6427 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:32:53.566190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.0.98.1:36975)
0926 09:32:53.566366 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.566367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.566401 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:32:53.566480 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.566480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.566491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.566536 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.566819 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51444: BlockingRecv error: recv got EOF from 127.0.0.1:51444 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260926 09:32:53.567018 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5cc)
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.568158 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.569160 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:53.576668 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.569696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.98.1:36975)
0926 09:32:53.569776 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.569778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.569809 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:32:53.569892 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.569893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.569903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.569951 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.570365 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.570908 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.571817 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.572060 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.572159 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.574244 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:53.574245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:53.574245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:53.574252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:53.575558 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.575905 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.575906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.575906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.575964 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.576177 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.576178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.576178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.576278 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:53.576386 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:53.576484 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 09:32:53.576572 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 09:32:53.578197 6427 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:32:53.577707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.0.98.1:36975)
0926 09:32:53.577741 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.577743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.577769 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:32:53.577815 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.577815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.577822 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.577869 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.578145 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51472: BlockingRecv error: recv got EOF from 127.0.0.1:51472 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:32:53.578294 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790415146110679-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.579226 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:53.580134 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[5916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:32:53.587149 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:53.580679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.0.98.1:36975)
0926 09:32:53.580736 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:53.580737 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:53.580830 (+ 93us) server_negotiation.cc:407] Connection header received
0926 09:32:53.580904 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:53.580905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:53.580914 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:53.580987 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:53.581352 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.581830 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.582534 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:53.582727 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:53.582809 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:53.584794 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:53.584795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:53.584796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:53.584803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:53.585979 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.586377 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.586378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.586379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.586435 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:53.586644 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:53.586645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:53.586645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:53.586742 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:53.586844 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:53.586982 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 09:32:53.587048 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:32:53.588073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 6023
I20260926 09:32:53.592828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 6157
I20260926 09:32:53.597424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 6291
I20260926 09:32:53.601924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 5929
2026-09-26T09:32:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1910 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[6440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[6440](info): set up 2 sockets
Sep 26 09:32:53 dist-test-slave-21h8 krb5kdc[6440](info): commencing operation
krb5kdc: starting...
W20260926 09:32:54.655581 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:32:54 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:32:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 09:32:54.778277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:54.859117 6453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:54.859386 6453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:54.859432 6453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:54.860534 6453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:54.860569 6453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:54.860582 6453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:54.860595 6453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:54.860607 6453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:54.861974 6453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:54.862154 6453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:54.862318 6453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:54.867197 6459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:54.867224 6461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:54.867251 6458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:54.867340 6453 server_base.cc:1034] running on GCE node
I20260926 09:32:54.867652 6453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:54.867944 6453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:54.869128 6453 hybrid_clock.cc:648] HybridClock initialized: now 1790415174869119 us; error 52 us; skew 500 ppm
Sep 26 09:32:54 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:54.871652 6453 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 09:32:54.872048 6453 webserver.cc:460] Webserver started at http://127.0.98.62:43907/ using document root <none> and password file <none>
I20260926 09:32:54.872229 6453 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:54.872269 6453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:54.872356 6453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:54.873394 6453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "5cf7ce0434d14c5283f588019c553a7a"
format_stamp: "Formatted at 2026-09-26 09:32:54 on dist-test-slave-21h8"
server_key: "c764e62758c323bb5a28455ec163a98f"
server_key_iv: "9e8c128d58697a448f20813ff89f93e4"
server_key_version: "encryptionkey@0"
I20260926 09:32:54.873703 6453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "5cf7ce0434d14c5283f588019c553a7a"
format_stamp: "Formatted at 2026-09-26 09:32:54 on dist-test-slave-21h8"
server_key: "c764e62758c323bb5a28455ec163a98f"
server_key_iv: "9e8c128d58697a448f20813ff89f93e4"
server_key_version: "encryptionkey@0"
I20260926 09:32:54.874951 6453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:32:54.875715 6468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:54.875870 6453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:54.875957 6453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "5cf7ce0434d14c5283f588019c553a7a"
format_stamp: "Formatted at 2026-09-26 09:32:54 on dist-test-slave-21h8"
server_key: "c764e62758c323bb5a28455ec163a98f"
server_key_iv: "9e8c128d58697a448f20813ff89f93e4"
server_key_version: "encryptionkey@0"
I20260926 09:32:54.876011 6453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:54.893551 6453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:54.895941 6453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:54.896062 6453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:54.899951 6453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39679
I20260926 09:32:54.900017 6520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39679 every 8 connection(s)
I20260926 09:32:54.900379 6453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:32:54.901381 6521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:54.903162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 6453
I20260926 09:32:54.903254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:32:54.903492 392 external_mini_cluster.cc:1428] Setting key ed4ecc0d72e9099170026f74eb4983a5
I20260926 09:32:54.903905 6521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: Bootstrap starting.
I20260926 09:32:54.904575 6521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:54.904896 6521 log.cc:824] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:54.905705 6521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: No bootstrap required, opened a new log
I20260926 09:32:54.907768 6521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260926 09:32:54.907939 6521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:54.907967 6521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf7ce0434d14c5283f588019c553a7a, State: Initialized, Role: FOLLOWER
I20260926 09:32:54.908124 6521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260926 09:32:54.908198 6521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:32:54.908221 6521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:32:54.908290 6521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:54.909015 6521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260926 09:32:54.909129 6521 leader_election.cc:302] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf7ce0434d14c5283f588019c553a7a; no voters:
I20260926 09:32:54.909299 6521 leader_election.cc:288] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:32:54.909361 6526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:54.909502 6526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 1 LEADER]: Becoming Leader. State: Replica: 5cf7ce0434d14c5283f588019c553a7a, State: Running, Role: LEADER
I20260926 09:32:54.909564 6521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:54.909605 6526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260926 09:32:54.909893 6526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf7ce0434d14c5283f588019c553a7a. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260926 09:32:54.909983 6526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:54.909902 6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260926 09:32:54.910120 6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
Sep 26 09:32:54 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
W20260926 09:32:54.911485 6542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:32:54.911585 6542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:32:54.911643 6531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:54.911818 6531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:54.913681 6531 catalog_manager.cc:1347] Generated new cluster ID: 0408c85ef29c4ec2a29f310ecd0cd43d
I20260926 09:32:54.913730 6531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:54.915490 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:54.904063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.0.98.62:39679)
0926 09:32:54.904247 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:54.904259 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:54.904286 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:32:54.904797 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:54.904806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:54.904859 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:54.904978 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:54.905806 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:54.906723 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:54.907842 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:54.908481 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:54.908563 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:54.911444 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:54.911448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:54.911448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:54.911460 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:54.913548 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:54.914004 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:54.914007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:54.914007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:54.914103 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:54.914273 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:54.914273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:54.914274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:54.914548 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:54.914669 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:54.915125 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 09:32:54.915250 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260926 09:32:54.936528 6531 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:32:54.937152 6531 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:54.942695 6531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: Generated new TSK 0
I20260926 09:32:54.942974 6531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 09:32:54.978998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:55.062760 6547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:55.062989 6547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:55.063015 6547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:55.064267 6547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:55.064316 6547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:55.064344 6547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:55.065879 6547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:55.066151 6547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:55.066430 6547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:55.071846 6555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.071846 6552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.071919 6553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.072214 6547 server_base.cc:1034] running on GCE node
I20260926 09:32:55.072530 6547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:55.072821 6547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:55.073993 6547 hybrid_clock.cc:648] HybridClock initialized: now 1790415175073973 us; error 44 us; skew 500 ppm
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:55.076838 6547 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 09:32:55.077375 6547 webserver.cc:460] Webserver started at http://127.0.98.1:39955/ using document root <none> and password file <none>
I20260926 09:32:55.077639 6547 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:55.077715 6547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:55.077870 6547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:55.079043 6547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "8002dfb7e8bd4a9097332207c63fe0b2"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "12257c2a079ad2a8fa9857d163d4f525"
server_key_iv: "4eb9f6a14669d34e2192ffb9c6b7fb68"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.079474 6547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8002dfb7e8bd4a9097332207c63fe0b2"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "12257c2a079ad2a8fa9857d163d4f525"
server_key_iv: "4eb9f6a14669d34e2192ffb9c6b7fb68"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.080948 6547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:55.081730 6562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.081956 6547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:55.082063 6547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "8002dfb7e8bd4a9097332207c63fe0b2"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "12257c2a079ad2a8fa9857d163d4f525"
server_key_iv: "4eb9f6a14669d34e2192ffb9c6b7fb68"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.082152 6547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:55.105679 6547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:55.107987 6547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:55.108181 6547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:55.108438 6547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:55.108783 6547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:55.108842 6547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.108886 6547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:55.108913 6547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.115453 6547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41441
I20260926 09:32:55.115500 6675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41441 every 8 connection(s)
I20260926 09:32:55.115851 6547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:32:55.124475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 6547
I20260926 09:32:55.124599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:32:55.124804 392 external_mini_cluster.cc:1428] Setting key 380f56002db0f882d0b27dfb49fedf0f
I20260926 09:32:55.125792 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.116638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33891 (local address 127.0.98.62:39679)
0926 09:32:55.116747 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.116750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.116939 (+ 189us) server_negotiation.cc:407] Connection header received
0926 09:32:55.117427 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.117428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.117446 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.117501 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.118461 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.118981 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.120108 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.120310 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.120376 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.122593 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.122594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.122595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.122601 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.124168 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.124713 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.124714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.124715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.124784 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.125110 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.125111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.125111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.125248 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.125316 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.125649 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.125727 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 09:32:55.126104 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.116589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:33891)
0926 09:32:55.116803 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.116841 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.117328 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.117559 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.117562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.118155 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.118353 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.118358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.119080 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.119081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.119997 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.120000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.120395 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.120396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.120506 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.121078 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.121086 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.122503 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.124312 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.124320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.124323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.124633 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.124864 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.124865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.124865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.125010 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.125316 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.125321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.125473 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.125613 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.125799 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":130,"threads_started":1}
I20260926 09:32:55.127279 6676 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 09:32:55.127422 6676 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:55.127614 6676 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 09:32:55.128185 6485 ts_manager.cc:194] Registered new tserver with Master: 8002dfb7e8bd4a9097332207c63fe0b2 (127.0.98.1:41441)
I20260926 09:32:55.129190 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33891
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 09:32:55.150738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:55.233467 6681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:55.233750 6681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:55.233798 6681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:55.235255 6681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:55.235378 6681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:55.235432 6681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:55.237111 6681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:55.237421 6681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:55.237673 6681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:55.243216 6687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.243261 6689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.243398 6681 server_base.cc:1034] running on GCE node
W20260926 09:32:55.243280 6686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.243693 6681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:55.244004 6681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:55.245177 6681 hybrid_clock.cc:648] HybridClock initialized: now 1790415175245159 us; error 54 us; skew 500 ppm
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:55.247864 6681 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 09:32:55.248311 6681 webserver.cc:460] Webserver started at http://127.0.98.2:44325/ using document root <none> and password file <none>
I20260926 09:32:55.248559 6681 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:55.248607 6681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:55.248754 6681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:55.249872 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "ab31df190ad31a1fddb23a94111acd6e"
server_key_iv: "8da30c48cbd9a6790ff61be7a77a8f98"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.250240 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "ab31df190ad31a1fddb23a94111acd6e"
server_key_iv: "8da30c48cbd9a6790ff61be7a77a8f98"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.251667 6681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:32:55.252561 6696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.252803 6681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:55.252907 6681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "ab31df190ad31a1fddb23a94111acd6e"
server_key_iv: "8da30c48cbd9a6790ff61be7a77a8f98"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.252991 6681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:55.264954 6681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:55.267505 6681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:55.267701 6681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:55.267971 6681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:55.268316 6681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:55.268378 6681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.268421 6681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:55.268455 6681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.274611 6681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35219
I20260926 09:32:55.274688 6809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35219 every 8 connection(s)
I20260926 09:32:55.275084 6681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:55.275767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 6681
I20260926 09:32:55.275935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:32:55.276156 392 external_mini_cluster.cc:1428] Setting key 811bf53320f93035f79810be3b30e744
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:32:55.286964 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.275955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41361 (local address 127.0.98.62:39679)
0926 09:32:55.276014 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.276017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.276299 (+ 282us) server_negotiation.cc:407] Connection header received
0926 09:32:55.276958 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.276960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.276972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.277034 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.278445 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.278951 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.280284 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.280519 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.280622 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.283445 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.283447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.283447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.283456 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.285406 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.285883 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.285885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.285885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.285949 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.286250 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.286251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.286251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.286398 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.286457 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.286759 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.286860 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:32:55.287178 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.275959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:41361)
0926 09:32:55.276165 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.276212 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.276805 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.277116 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.277121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.278020 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.278342 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.278349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.279126 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.279127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.280136 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.280144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.280634 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.280635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.280781 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.281395 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.281404 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.283258 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.285512 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.285523 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.285525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.285777 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.286044 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.286045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.286046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.286173 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.286459 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.286466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.286621 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.286742 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.286919 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260926 09:32:55.287755 6810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 09:32:55.287936 6810 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:55.288160 6810 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 09:32:55.288663 6485 ts_manager.cc:194] Registered new tserver with Master: ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219)
I20260926 09:32:55.289124 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 09:32:55.307966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:55.401155 6815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:55.401376 6815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:55.401401 6815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:55.402621 6815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:55.402669 6815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:55.402734 6815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:55.404377 6815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:55.404650 6815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:55.404907 6815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:55.410218 6821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.410226 6823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.410403 6815 server_base.cc:1034] running on GCE node
W20260926 09:32:55.410616 6820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.410864 6815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:55.411212 6815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:55.412410 6815 hybrid_clock.cc:648] HybridClock initialized: now 1790415175412389 us; error 37 us; skew 500 ppm
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:55.415277 6815 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 09:32:55.415841 6815 webserver.cc:460] Webserver started at http://127.0.98.3:42343/ using document root <none> and password file <none>
I20260926 09:32:55.416116 6815 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:55.416167 6815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:55.416363 6815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:55.417541 6815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "557dde2da1f84cb39c69b82507192090"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "d6627f52cc06f8f359f6c6239acd379c"
server_key_iv: "0e6c6d00f1f828852326523c0f549c24"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.417912 6815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "557dde2da1f84cb39c69b82507192090"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "d6627f52cc06f8f359f6c6239acd379c"
server_key_iv: "0e6c6d00f1f828852326523c0f549c24"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.419291 6815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:32:55.420142 6830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.420298 6815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:55.420387 6815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "557dde2da1f84cb39c69b82507192090"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "d6627f52cc06f8f359f6c6239acd379c"
server_key_iv: "0e6c6d00f1f828852326523c0f549c24"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.420444 6815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:55.431752 6815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:55.434026 6815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:55.434203 6815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:55.434450 6815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:55.434780 6815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:55.434839 6815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.434883 6815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:55.434907 6815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.440824 6815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40099
I20260926 09:32:55.440912 6943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40099 every 8 connection(s)
I20260926 09:32:55.441258 6815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:32:55.443754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 6815
I20260926 09:32:55.443832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:32:55.444028 392 external_mini_cluster.cc:1428] Setting key fc485578e62cd2d973dcec09b0e71db6
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:32:55.450733 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.442048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49547 (local address 127.0.98.62:39679)
0926 09:32:55.442115 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.442117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.442310 (+ 193us) server_negotiation.cc:407] Connection header received
0926 09:32:55.442896 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.442898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.442908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.442948 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.444101 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.444701 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.445778 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.445998 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.446057 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.448306 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.448307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.448307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.448313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.449573 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.449940 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.449941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.449941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.449993 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.450217 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.450217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.450218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.450316 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.450369 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.450609 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.450668 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:32:55.450961 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.442055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:49547)
0926 09:32:55.442202 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.442239 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.442802 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.443013 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.443016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.443722 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.443994 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.444000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.444775 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.444776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.445692 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.445694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.446093 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.446094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.446196 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.446785 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.446794 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.448218 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.449650 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.449655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.449657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.449868 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.450063 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.450064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.450064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.450158 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.450378 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.450382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.450495 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.450581 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.450699 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260926 09:32:55.451658 6944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 09:32:55.451761 6944 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:55.451905 6944 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 09:32:55.452277 6485 ts_manager.cc:194] Registered new tserver with Master: 557dde2da1f84cb39c69b82507192090 (127.0.98.3:40099)
I20260926 09:32:55.452613 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49547
I20260926 09:32:55.455909 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 09:32:55.482395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41787
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39679,127.0.98.61:41787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:55.565663 6949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:55.565940 6949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:55.565966 6949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:55.567147 6949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:55.567198 6949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:55.567214 6949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:55.567227 6949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:55.567241 6949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:55.568830 6949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39679,127.0.98.61:41787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
W20260926 09:32:55.568953 6949 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:39679,127.0.98.61:41787'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 09:32:55.569152 6949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:55.569442 6949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:55.574731 6955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.574746 6954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.574767 6957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.575052 6949 server_base.cc:1034] running on GCE node
I20260926 09:32:55.575295 6949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:55.575680 6949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:55.576880 6949 hybrid_clock.cc:648] HybridClock initialized: now 1790415175576877 us; error 39 us; skew 500 ppm
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:55.579655 6949 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 09:32:55.580080 6949 webserver.cc:460] Webserver started at http://127.0.98.61:34251/ using document root <none> and password file <none>
I20260926 09:32:55.580272 6949 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:55.580312 6949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:55.580412 6949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:55.581458 6949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data/instance:
uuid: "3aba30e4d364422c91a0b571d7d3c8e6"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "0afb4bf330dd8f410e4c551d1efc8f78"
server_key_iv: "75583b1122e50da9fed91dbbca58c585"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.581768 6949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal/instance:
uuid: "3aba30e4d364422c91a0b571d7d3c8e6"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "0afb4bf330dd8f410e4c551d1efc8f78"
server_key_iv: "75583b1122e50da9fed91dbbca58c585"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.583084 6949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:32:55.583878 6964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.584079 6949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:32:55.584167 6949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
uuid: "3aba30e4d364422c91a0b571d7d3c8e6"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "0afb4bf330dd8f410e4c551d1efc8f78"
server_key_iv: "75583b1122e50da9fed91dbbca58c585"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.584224 6949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:55.591516 6949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:55.593484 6949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:55.593612 6949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:55.597534 6949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41787
I20260926 09:32:55.597568 7016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41787 every 8 connection(s)
I20260926 09:32:55.597975 6949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:32:55.599035 7017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:55.600657 7017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:32:55.607424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 6949
I20260926 09:32:55.610702 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.601455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.0.98.62:39679)
0926 09:32:55.601548 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.601551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.601731 (+ 180us) server_negotiation.cc:407] Connection header received
0926 09:32:55.602195 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.602196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.602205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.602259 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.602861 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.603430 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.604421 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.604613 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.604663 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.607357 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.607359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.607359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.607368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.609183 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.609600 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.609602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.609602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.609667 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.609894 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.609895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.609896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.610091 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.610155 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.610510 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.610597 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 09:32:55.610935 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.601396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37028)
0926 09:32:55.601598 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.601645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.602110 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.602318 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.602321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.602588 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.602789 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.602792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.603507 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.603508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.604344 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.604346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.604685 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.604686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.604786 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.605384 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.605395 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.607223 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.609267 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.609273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.609274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.609518 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.609729 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.609730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.609730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.609830 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.610188 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.610192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.610365 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.610482 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.610691 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260926 09:32:55.611707 7017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:32:55.617415 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.607963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.0.98.61:41787)
0926 09:32:55.608164 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.608169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.608189 (+ 20us) server_negotiation.cc:407] Connection header received
0926 09:32:55.608248 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.608250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.608264 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.608329 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.609000 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.610048 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.610900 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.611438 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.611510 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.614220 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.614229 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.614229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.614236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.615974 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.616381 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.616385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.616385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.616457 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.616695 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.616696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.616696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.616887 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.617118 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.617176 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.617274 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
I20260926 09:32:55.619380 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.611980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.0.1:36752)
0926 09:32:55.612039 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.612050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.612089 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.612349 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.612350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.612505 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.612568 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.612570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.613216 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.613217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.613696 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.613697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.614220 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.614221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.614296 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.614672 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.614676 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.616168 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.618553 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.618554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.618554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.618711 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.618883 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.618883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.618884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.618931 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.619105 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.619105 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.619138 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.619187 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.619260 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 09:32:55.619545 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.612064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.0.98.61:41787)
0926 09:32:55.612214 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.612216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.612223 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:32:55.612244 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.612245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.612259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.612317 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.612650 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.613126 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.613799 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.614146 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.614210 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.616247 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.616248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.616248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.616253 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.618482 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.618777 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.618778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.618778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.618827 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.618967 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.618968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.618968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.619054 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.619098 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.619202 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.619303 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260926 09:32:55.620810 7017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Bootstrap starting.
I20260926 09:32:55.621668 7017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:55.621974 7017 log.cc:824] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:55.622805 7017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: No bootstrap required, opened a new log
I20260926 09:32:55.624495 7017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } }
I20260926 09:32:55.624648 7017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:55.624696 7017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aba30e4d364422c91a0b571d7d3c8e6, State: Initialized, Role: FOLLOWER
I20260926 09:32:55.624856 7017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } }
I20260926 09:32:55.625200 7017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:55.625170 7024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } } }
I20260926 09:32:55.625314 7024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:32:55.626902 7035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:32:55.626956 7035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:32:55.628865 6949 master_runner.cc:428] Detected that this master 3aba30e4d364422c91a0b571d7d3c8e6 is joining an existing cluster
I20260926 09:32:55.628913 6949 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41787
I20260926 09:32:55.637445 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.629910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37038)
0926 09:32:55.630078 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.630094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.630180 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.630509 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.630509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.630734 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.630796 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.630798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.631731 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.631732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.632639 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.632642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.633081 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.633082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.633151 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.633668 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.633674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.634135 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.636130 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.636130 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.636131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.636386 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.636628 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.636629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.636630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.636695 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.637054 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.637055 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.637103 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.637220 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.637323 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260926 09:32:55.637435 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.629955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.0.98.62:39679)
0926 09:32:55.630001 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.630003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.630167 (+ 164us) server_negotiation.cc:407] Connection header received
0926 09:32:55.630406 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.630407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.630418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.630468 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.630864 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.631640 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.632742 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.632926 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.633068 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.634223 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.634224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.634225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.634232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.636046 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.636476 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.636477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.636478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.636547 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.636776 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.636777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.636777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.636967 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.637055 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.637259 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.637333 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:32:55.638195 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.629932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.0.98.61:41787)
0926 09:32:55.629973 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.629975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.630267 (+ 292us) server_negotiation.cc:407] Connection header received
0926 09:32:55.630291 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.630292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.630302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.630357 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.630804 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.631654 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.633679 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.633981 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.634054 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.635169 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.635170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.635170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.635177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.636812 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.637197 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.637198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.637198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.637255 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.637466 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.637467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.637467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.637578 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.637620 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.638018 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.638110 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:55.638593 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.630014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.0.1:36754)
0926 09:32:55.630166 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.630176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.630227 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.630420 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.630421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.630639 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.630712 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.630713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.632757 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.632758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.633602 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.633604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.634078 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.634079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.634145 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.634612 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.634618 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.635056 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.636899 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.636899 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.636900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.637114 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.637328 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.637329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.637329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.637403 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.637873 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.637874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.637913 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.638402 (+ 489us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.638491 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:32:55.647434 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.639828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.0.1:36770)
0926 09:32:55.640095 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.640149 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.640229 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.640363 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.640366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.640706 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.640772 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.640774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.641397 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.641398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.642356 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.642359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.642692 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.642694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.642783 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.643195 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.643201 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.644698 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.646252 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.646260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.646262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.646478 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.646692 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.646693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.646694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.646761 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.647029 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.647036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.647087 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.647195 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.647311 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
I20260926 09:32:55.647432 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.639914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.0.98.61:41787)
0926 09:32:55.639977 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.639980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.640225 (+ 245us) server_negotiation.cc:407] Connection header received
0926 09:32:55.640303 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.640303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.640311 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.640356 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.640846 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.641314 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.642441 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.642614 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.642665 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.644796 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.644797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.644798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.644805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.646160 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.646560 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.646561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.646562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.646622 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.646828 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.646829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.646830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.646943 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.647023 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.647218 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.647300 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:32:55.647984 6485 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41787: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41787
}
attrs {
promote: true
}
}
dest_uuid: "5cf7ce0434d14c5283f588019c553a7a"
cas_config_opid_index: -1
I20260926 09:32:55.648312 6485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } }
I20260926 09:32:55.648825 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3aba30e4d364422c91a0b571d7d3c8e6 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } }
I20260926 09:32:55.649142 6991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:55.649973 7045 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41787
I20260926 09:32:55.650063 6991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Refusing update from remote peer 5cf7ce0434d14c5283f588019c553a7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 09:32:55.650208 6949 master.cc:537] Master@127.0.98.61:41787 shutting down...
I20260926 09:32:55.650447 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:32:55.651161 6471 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 09:32:55.651721 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } }
I20260926 09:32:55.651818 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
W20260926 09:32:55.651825 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a -> Peer 3aba30e4d364422c91a0b571d7d3c8e6 (127.0.98.61:41787): Couldn't send request to peer 3aba30e4d364422c91a0b571d7d3c8e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 09:32:55.653211 6949 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:55.653301 6949 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:55.653374 6949 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: stopping tablet replica
I20260926 09:32:55.666369 6949 master.cc:552] Master@127.0.98.61:41787 shutdown complete.
I20260926 09:32:55.666440 6949 master_runner.cc:305] Clearing existing system tablet
I20260926 09:32:55.666590 6949 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:55.666642 6949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:55.667495 6949 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:32:55.668051 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.668227 6949 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:55.668293 6949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.668341 6949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
uuid: "3aba30e4d364422c91a0b571d7d3c8e6"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "0afb4bf330dd8f410e4c551d1efc8f78"
server_key_iv: "75583b1122e50da9fed91dbbca58c585"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.669323 6949 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:55.671098 6949 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 09:32:55.671186 6949 log.cc:1196] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 09:32:55.671452 6949 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Deleting consensus metadata
I20260926 09:32:55.671603 6949 master_runner.cc:315] Copying system tablet from 127.0.98.62:39679
I20260926 09:32:55.672248 6949 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39679
I20260926 09:32:55.679090 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.672587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37044)
0926 09:32:55.672717 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.672728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.672799 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.672959 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.672960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.673176 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.673271 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.673273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.674278 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.674279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.674844 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.674846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.675382 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.675383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.675454 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.676004 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.676009 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.676305 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.678068 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.678069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.678069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.678259 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.678466 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.678467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.678479 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.678539 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.678818 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.678818 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.678863 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.678918 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.679006 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":91,"threads_started":1}
I20260926 09:32:55.679155 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.672658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.0.98.62:39679)
0926 09:32:55.672714 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.672716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.672780 (+ 64us) server_negotiation.cc:407] Connection header received
0926 09:32:55.672874 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.672875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.672885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.672941 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.673387 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.674189 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.674937 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.675254 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.675364 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.676399 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.676401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.676401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.676409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.677979 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.678332 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.678333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.678333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.678398 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.678593 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.678594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.678594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.678727 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.678814 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.678970 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.679060 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:32:55.679481 6505 tablet_copy_service.cc:140] P 5cf7ce0434d14c5283f588019c553a7a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3aba30e4d364422c91a0b571d7d3c8e6 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37044)
I20260926 09:32:55.679587 6505 tablet_copy_service.cc:161] P 5cf7ce0434d14c5283f588019c553a7a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3aba30e4d364422c91a0b571d7d3c8e6 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37044: session id = 3aba30e4d364422c91a0b571d7d3c8e6-00000000000000000000000000000000
I20260926 09:32:55.680486 6505 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 09:32:55.681200 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 09:32:55.682541 6949 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: tablet copy: Starting download of 0 data blocks...
I20260926 09:32:55.682706 6949 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: tablet copy: Starting download of 1 WAL segments...
I20260926 09:32:55.683669 6949 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 09:32:55.684525 6505 tablet_copy_service.cc:342] P 5cf7ce0434d14c5283f588019c553a7a: Request end of tablet copy session 3aba30e4d364422c91a0b571d7d3c8e6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37044
I20260926 09:32:55.684594 6505 tablet_copy_service.cc:433] P 5cf7ce0434d14c5283f588019c553a7a: ending tablet copy session 3aba30e4d364422c91a0b571d7d3c8e6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3aba30e4d364422c91a0b571d7d3c8e6
I20260926 09:32:55.689160 6949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:55.690487 7061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.690591 6949 server_base.cc:1034] running on GCE node
W20260926 09:32:55.690707 7058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.690939 7059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.691118 6949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:55.691298 6949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:55.692456 6949 hybrid_clock.cc:648] HybridClock initialized: now 1790415175692430 us; error 44 us; skew 500 ppm
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:55.693895 6949 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 09:32:55.694175 6949 webserver.cc:460] Webserver started at http://127.0.98.61:34675/ using document root <none> and password file <none>
I20260926 09:32:55.694319 6949 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:55.694368 6949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:55.695027 6949 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:55.695662 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.695794 6949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.695864 6949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
uuid: "3aba30e4d364422c91a0b571d7d3c8e6"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "0afb4bf330dd8f410e4c551d1efc8f78"
server_key_iv: "75583b1122e50da9fed91dbbca58c585"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.695947 6949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:55.701619 6949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:55.703267 6949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:55.706480 6949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41787
I20260926 09:32:55.706887 6949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:32:55.706929 7120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41787 every 8 connection(s)
I20260926 09:32:55.707834 7121 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:32:55.708359 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Bootstrap starting.
I20260926 09:32:55.711490 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:32:55.711755 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: Bootstrap complete.
I20260926 09:32:55.711967 7121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } }
I20260926 09:32:55.712076 7121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3aba30e4d364422c91a0b571d7d3c8e6, State: Initialized, Role: LEARNER
I20260926 09:32:55.712126 7121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } }
I20260926 09:32:55.712320 7121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:55.712311 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } }
I20260926 09:32:55.712421 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:32:55.713933 7135 catalog_manager.cc:1259] Loaded cluster ID: 0408c85ef29c4ec2a29f310ecd0cd43d
I20260926 09:32:55.714004 7135 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: loading cluster ID for follower catalog manager: success
I20260926 09:32:55.715036 7135 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: acquiring CA information for follower catalog manager: success
I20260926 09:32:55.715364 7135 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:32:55.726190 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.719168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.0.98.61:41787)
0926 09:32:55.719287 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.719289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.719297 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:32:55.719383 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.719383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.719394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.719483 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.720022 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.720688 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.721497 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.721712 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.721788 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.723187 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.723189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.723189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.723196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.724749 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.725106 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.725107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.725107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.725164 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.725372 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.725373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.725374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.725492 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.725989 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.726029 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.726098 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 09:32:55.754992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36145
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39679,127.0.98.61:41787,127.0.98.60:36145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:32:55.847041 7139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:55.847298 7139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:55.847335 7139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:55.848591 7139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:32:55.848670 7139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:55.848697 7139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:55.848730 7139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:32:55.848752 7139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:32:55.850417 7139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39679,127.0.98.61:41787,127.0.98.60:36145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:55.850703 7139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:55.850953 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:55.856492 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.856492 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.856552 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.856839 7139 server_base.cc:1034] running on GCE node
I20260926 09:32:55.857123 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:55.857438 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:55.858618 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790415175858600 us; error 30 us; skew 500 ppm
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:55.861282 7139 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 09:32:55.861776 7139 webserver.cc:460] Webserver started at http://127.0.98.60:44437/ using document root <none> and password file <none>
I20260926 09:32:55.861989 7139 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:55.862032 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:55.862128 7139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:32:55.863185 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data/instance:
uuid: "78b19be33668429f9b8e7fa432582542"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "2617648e972d1c6e64b1a02cb04b7a7a"
server_key_iv: "6a9ae2a95b0dd6202a6b649eaa301a05"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.863582 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal/instance:
uuid: "78b19be33668429f9b8e7fa432582542"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "2617648e972d1c6e64b1a02cb04b7a7a"
server_key_iv: "6a9ae2a95b0dd6202a6b649eaa301a05"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.864998 7139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:55.865685 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.865845 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:55.865935 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
uuid: "78b19be33668429f9b8e7fa432582542"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "2617648e972d1c6e64b1a02cb04b7a7a"
server_key_iv: "6a9ae2a95b0dd6202a6b649eaa301a05"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.865994 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:55.872967 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:55.874850 7139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:55.874963 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:55.879405 7139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36145
I20260926 09:32:55.879444 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36145 every 8 connection(s)
I20260926 09:32:55.879828 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:32:55.880713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 7139
I20260926 09:32:55.880893 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:55.883605 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:32:55.892768 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.881402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.98.60:36145)
0926 09:32:55.881618 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.881626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.881652 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:32:55.882058 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.882065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.882112 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.882231 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.883054 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.884100 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.885000 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.885669 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.885785 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.888747 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.888754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.888755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.888762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.890943 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.891296 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.891300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.891301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.891452 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.891698 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.891699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.891699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.891983 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.892127 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.892340 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.892501 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
I20260926 09:32:55.892810 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.884460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.0.98.62:39679)
0926 09:32:55.884513 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.884515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.884666 (+ 151us) server_negotiation.cc:407] Connection header received
0926 09:32:55.884737 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.884739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.884749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.884844 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.885355 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.886041 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.887421 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.887674 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.887755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.890080 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.890081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.890082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.890090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.891597 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.891958 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.891959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.891959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.892020 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.892226 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.892227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.892227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.892379 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.892447 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.892680 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.892745 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:32:55.892879 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.884293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37058)
0926 09:32:55.884553 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.884592 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.884685 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.884876 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.884879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.885174 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.885237 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.885238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.886160 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.886161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.887272 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.887275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.887775 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.887776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.887838 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.888371 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.888382 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.889983 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.891666 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.891673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.891675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.891882 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.892087 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.892088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.892088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.892156 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.892476 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.892481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.892536 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.892634 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.892733 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
I20260926 09:32:55.893347 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:32:55.900637 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.893684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.0.98.61:41787)
0926 09:32:55.893751 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.893753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.893762 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:55.893787 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.893788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.893803 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.893852 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.894100 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.894601 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.895241 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.895530 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.895588 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.897502 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.897504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.897504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.897512 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.899285 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.899763 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.899764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.899765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.899887 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.900136 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.900136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.900137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.900262 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.900331 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.900488 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.900568 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:32:55.900667 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.893583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.0.1:36800)
0926 09:32:55.893636 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.893647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.893691 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.893854 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.893854 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.893965 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.894009 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.894010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.894675 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.894676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.895166 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.895167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.895598 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.895598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.895650 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.896063 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.896067 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.897342 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.899481 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.899482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.899483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.899689 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.899986 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.899987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.899987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.900065 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.900338 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.900339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.900385 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.900466 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.900579 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:32:55.901146 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415175, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 09:32:55.908535 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.901589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.0.98.60:36145)
0926 09:32:55.901661 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.901664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.901674 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:55.901746 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.901746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.901758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.901824 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.902144 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.902738 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.903601 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.903875 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.903982 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.906041 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.906042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.906043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.906050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.907283 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.907691 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.907692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.907692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.907748 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.907933 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.907934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.907935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.908052 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.908129 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.908275 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.908424 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":52}
I20260926 09:32:55.908535 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.901506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.0.1:43414)
0926 09:32:55.901573 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.901586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.901673 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.901827 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.901828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.901987 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.902030 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.902031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.902858 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.902859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.903488 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.903491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.904026 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.904027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.904092 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.904622 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.904628 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.905944 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.907438 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.907439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.907439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.907607 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.907812 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.907813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.907813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.907870 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.908120 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.908121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.908157 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.908252 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.908378 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":2}
I20260926 09:32:55.910162 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Bootstrap starting.
I20260926 09:32:55.911522 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:55.911818 7207 log.cc:824] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:55.912585 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: No bootstrap required, opened a new log
I20260926 09:32:55.914180 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } }
I20260926 09:32:55.914317 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:55.914340 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b19be33668429f9b8e7fa432582542, State: Initialized, Role: FOLLOWER
I20260926 09:32:55.914525 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } }
I20260926 09:32:55.914799 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } } }
I20260926 09:32:55.914853 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:55.914880 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:32:55.916706 7223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:32:55.916770 7223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:32:55.919106 7139 master_runner.cc:428] Detected that this master 78b19be33668429f9b8e7fa432582542 is joining an existing cluster
I20260926 09:32:55.919165 7139 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36145
I20260926 09:32:55.928248 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.920360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.0.98.61:41787)
0926 09:32:55.920410 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.920413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.920590 (+ 177us) server_negotiation.cc:407] Connection header received
0926 09:32:55.920685 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.920686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.920698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.920728 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.921362 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.922147 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923291 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.923624 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923668 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.924863 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.924864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.924864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.924871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.926462 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.926950 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.926951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.926952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.927022 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.927259 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.927260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.927260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.927479 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.927547 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.928096 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.928175 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:32:55.928273 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.920328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.0.1:36806)
0926 09:32:55.920525 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.920536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.920615 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.920798 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.920799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.921193 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.921268 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.921270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.922276 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.922277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.923193 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.923195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.923747 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.923811 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.924286 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.924291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.924757 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.926545 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.926546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.926547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.926826 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.927109 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.927110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.927110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.927183 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.927910 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.927911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.927963 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.928052 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.928165 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260926 09:32:55.944167 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.920372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.98.60:36145)
0926 09:32:55.920410 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.920413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.920589 (+ 176us) server_negotiation.cc:407] Connection header received
0926 09:32:55.920682 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.920683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.920693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.920722 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.921350 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.922095 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923245 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.923541 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923581 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.941112 (+ 17531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.941114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.941114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.941121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.942818 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.943389 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.943390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.943391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.943445 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.943631 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.943631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.943632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.943759 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.943800 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.943982 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.944066 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:32:55.944228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 6547
I20260926 09:32:55.944433 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.920616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.0.98.62:39679)
0926 09:32:55.920660 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.920662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.920892 (+ 230us) server_negotiation.cc:407] Connection header received
0926 09:32:55.920986 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.920987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.920999 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.921031 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.921429 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.922177 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923251 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.923580 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923619 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.940780 (+ 17161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.940783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.940783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.940794 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.942488 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.943141 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.943143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.943143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.943213 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.943603 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.943604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.943605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.943829 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.943913 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.944168 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.944271 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:55.944638 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.920418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.0.1:43422)
0926 09:32:55.920529 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.920539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.920613 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.920787 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.920787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.921181 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.921267 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.921269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.922184 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.922185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.923155 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.923157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.923666 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.923726 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.939888 (+ 16162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.939894 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.940629 (+ 735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.942996 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.942997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.942998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.943237 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.943515 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.943516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.943516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.943568 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.943838 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.943839 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.943876 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.944261 (+ 385us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.944357 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260926 09:32:55.945655 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.920723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37066)
0926 09:32:55.920835 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.920845 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.920915 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.921098 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.921098 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.921291 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.921350 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.921351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.922263 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.922263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.923155 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.923157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.923687 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.923687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.923740 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.940059 (+ 16319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.940064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.940629 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.942627 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.942628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.942629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.943032 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.943355 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.943356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.943357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.943439 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.943951 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.943951 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.944020 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.945312 (+ 1292us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.945465 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":39,"thread_start_us":62,"threads_started":1}
I20260926 09:32:55.953351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41441
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=39955
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787,127.0.98.60:36145 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:32:55 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 09:32:55.957640 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.948612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.0.1:43424)
0926 09:32:55.948672 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.948684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.948720 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.948904 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.948905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.949096 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.949151 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.949153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.950302 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.950304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.951157 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.951160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.951834 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.951835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.951912 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.952426 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.952432 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.954400 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.956422 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.956423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.956423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.956657 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.956895 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.956896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.956896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.956964 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.957295 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.957295 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.957348 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.957435 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.957543 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:32:55.958112 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.948693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.0.98.60:36145)
0926 09:32:55.948741 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.948743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.948755 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:32:55.948804 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.948805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.948818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.948893 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.949256 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.950197 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.951305 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.951689 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.951770 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.954512 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.954514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.954515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.954524 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.956326 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.956744 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.956745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.956746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.956813 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.957043 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.957044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.957044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.957211 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.957292 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.957488 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.957587 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:32:55.958946 6485 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36145: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "78b19be33668429f9b8e7fa432582542"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36145
}
attrs {
promote: true
}
}
dest_uuid: "5cf7ce0434d14c5283f588019c553a7a"
cas_config_opid_index: 5
I20260926 09:32:55.959256 6485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } }
I20260926 09:32:55.959832 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 78b19be33668429f9b8e7fa432582542 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.960886 7047 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36145
I20260926 09:32:55.960959 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.961047 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:55.961534 7139 master.cc:537] Master@127.0.98.60:36145 shutting down...
I20260926 09:32:55.961752 7181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:55.962731 7181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Refusing update from remote peer 5cf7ce0434d14c5283f588019c553a7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 09:32:55.963033 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:32:55.964510 6469 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a -> Peer 78b19be33668429f9b8e7fa432582542 (127.0.98.60:36145): Couldn't send request to peer 78b19be33668429f9b8e7fa432582542. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 09:32:55.965277 7139 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:55.965372 7139 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:55.965416 7139 tablet_replica.cc:330] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: stopping tablet replica
I20260926 09:32:55.970631 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.961247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.0.1:36812)
0926 09:32:55.964594 (+ 3347us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.964607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.964661 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.965022 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.965024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.965027 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:55.965086 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.965088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.967637 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.967639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.969200 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.969205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.970337 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.970338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.970411 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.970448 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.970499 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.970548 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3328}
I20260926 09:32:55.971127 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.961339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.0.98.61:41787)
0926 09:32:55.961390 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.961392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.964720 (+ 3328us) server_negotiation.cc:407] Connection header received
0926 09:32:55.964754 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.964756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.964783 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.964852 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:55.965205 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.967250 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.969325 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.970150 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.970222 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.970301 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.970910 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.971062 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:32:55.971349 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 LEARNER]: Refusing update from remote peer 5cf7ce0434d14c5283f588019c553a7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 09:32:55.971580 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 09:32:55.972163 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 78b19be33668429f9b8e7fa432582542 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.973537 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf7ce0434d14c5283f588019c553a7a. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.973637 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:32:55.973676 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: true } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.973718 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:32:55.973703 7046 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: attempting to promote NON_VOTER 3aba30e4d364422c91a0b571d7d3c8e6 to VOTER
I20260926 09:32:55.973918 7046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } }
I20260926 09:32:55.974423 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.974319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.0.1:43432)
0926 09:32:55.974358 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.974395 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 09:32:55.974625 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 LEARNER]: Refusing update from remote peer 5cf7ce0434d14c5283f588019c553a7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 09:32:55.974781 7047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 09:32:55.975354 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3aba30e4d364422c91a0b571d7d3c8e6 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.976343 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.976413 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:55.976466 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.976507 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:55.976732 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3aba30e4d364422c91a0b571d7d3c8e6 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.978468 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.978344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.0.1:43438)
0926 09:32:55.978402 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.978428 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 09:32:55.978536 6469 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a -> Peer 78b19be33668429f9b8e7fa432582542 (127.0.98.60:36145): Couldn't send request to peer 78b19be33668429f9b8e7fa432582542. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 09:32:55.978612 7234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:55.978713 7234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:32:55.978830 7139 master.cc:552] Master@127.0.98.60:36145 shutdown complete.
I20260926 09:32:55.978883 7139 master_runner.cc:305] Clearing existing system tablet
I20260926 09:32:55.979038 7139 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:55.979074 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:55.979803 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:32:55.980332 7236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:55.980499 7139 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:55.980557 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:32:55.980587 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
uuid: "78b19be33668429f9b8e7fa432582542"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "2617648e972d1c6e64b1a02cb04b7a7a"
server_key_iv: "6a9ae2a95b0dd6202a6b649eaa301a05"
server_key_version: "encryptionkey@0"
I20260926 09:32:55.981262 7139 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:55.982903 7139 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 09:32:55.983026 7139 log.cc:1196] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 09:32:55.983227 7139 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Deleting consensus metadata
I20260926 09:32:55.983481 7139 master_runner.cc:315] Copying system tablet from 127.0.98.62:39679
I20260926 09:32:55.984136 7139 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39679
I20260926 09:32:55.991096 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.984536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.0.98.62:39679)
0926 09:32:55.984586 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:55.984589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:55.984730 (+ 141us) server_negotiation.cc:407] Connection header received
0926 09:32:55.984811 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:55.984812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:55.984823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:55.984869 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:55.985227 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.985743 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.986503 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.986730 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.986816 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.988244 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:55.988246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:55.988246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:55.988256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:55.989860 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.990258 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.990260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.990260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.990327 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:55.990535 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:55.990536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:55.990536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:55.990663 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:55.990746 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:55.990910 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:32:55.990995 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:32:55.991097 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:55.984496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37082)
0926 09:32:55.984640 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:55.984651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:55.984745 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:55.984894 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:55.984895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:55.985073 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:55.985137 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.985138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.985839 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.985840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.986403 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:55.986405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:55.986814 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:55.986815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:55.986881 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:55.987788 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:55.987793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:55.988107 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:55.989977 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.989978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.989979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.990164 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.990396 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:55.990397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:55.990397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:55.990456 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:55.990736 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:55.990737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:55.990786 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 09:32:55.990880 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 09:32:55.991000 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260926 09:32:55.991511 6505 tablet_copy_service.cc:140] P 5cf7ce0434d14c5283f588019c553a7a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 78b19be33668429f9b8e7fa432582542 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37082)
I20260926 09:32:55.991585 6505 tablet_copy_service.cc:161] P 5cf7ce0434d14c5283f588019c553a7a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 78b19be33668429f9b8e7fa432582542 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37082: session id = 78b19be33668429f9b8e7fa432582542-00000000000000000000000000000000
I20260926 09:32:55.992039 6505 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 09:32:55.992544 7139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 09:32:55.993535 7139 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: tablet copy: Starting download of 0 data blocks...
I20260926 09:32:55.993712 7139 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: tablet copy: Starting download of 1 WAL segments...
I20260926 09:32:55.994436 7139 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 09:32:55.995219 6505 tablet_copy_service.cc:342] P 5cf7ce0434d14c5283f588019c553a7a: Request end of tablet copy session 78b19be33668429f9b8e7fa432582542-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37082
I20260926 09:32:55.995277 6505 tablet_copy_service.cc:433] P 5cf7ce0434d14c5283f588019c553a7a: ending tablet copy session 78b19be33668429f9b8e7fa432582542-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 78b19be33668429f9b8e7fa432582542
I20260926 09:32:55.997864 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:55.999130 7243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:55.999264 7246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:55.999413 7139 server_base.cc:1034] running on GCE node
W20260926 09:32:56.003526 7244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:56.003727 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:56.003917 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:56.005048 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790415176005045 us; error 29 us; skew 500 ppm
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:56.006505 7139 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 09:32:56.006829 7139 webserver.cc:460] Webserver started at http://127.0.98.60:37583/ using document root <none> and password file <none>
I20260926 09:32:56.006953 7139 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:56.007023 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:56.007754 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:56.008314 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.008522 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:32:56.008567 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
uuid: "78b19be33668429f9b8e7fa432582542"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "2617648e972d1c6e64b1a02cb04b7a7a"
server_key_iv: "6a9ae2a95b0dd6202a6b649eaa301a05"
server_key_version: "encryptionkey@0"
I20260926 09:32:56.008630 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:56.016659 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:56.018481 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:56.021740 7139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36145
I20260926 09:32:56.022101 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:32:56.022951 7306 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:32:56.023245 7305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36145 every 8 connection(s)
I20260926 09:32:56.023698 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Bootstrap starting.
I20260926 09:32:56.026383 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:32:56.026649 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Bootstrap complete.
I20260926 09:32:56.026872 7306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } }
I20260926 09:32:56.026975 7306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 78b19be33668429f9b8e7fa432582542, State: Initialized, Role: LEARNER
I20260926 09:32:56.027024 7306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } }
I20260926 09:32:56.027220 7306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:32:56.027213 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:56.027446 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:32:56.028734 7320 catalog_manager.cc:1259] Loaded cluster ID: 0408c85ef29c4ec2a29f310ecd0cd43d
I20260926 09:32:56.028823 7320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: loading cluster ID for follower catalog manager: success
I20260926 09:32:56.029978 7320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: acquiring CA information for follower catalog manager: success
I20260926 09:32:56.030274 7320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 09:32:56.077921 7232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:56.078256 7232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:56.078297 7232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:56.080111 7232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:56.080176 7232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:56.080219 7232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:32:56.082376 7232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39955
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787,127.0.98.60:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:56.082639 7232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:56.082901 7232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:56.089120 7325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:56.089124 7328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:56.089124 7326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:56.089556 7232 server_base.cc:1034] running on GCE node
I20260926 09:32:56.089774 7232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:56.090121 7232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:56.091352 7232 hybrid_clock.cc:648] HybridClock initialized: now 1790415176091335 us; error 32 us; skew 500 ppm
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:56.095075 7232 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 09:32:56.095779 7232 webserver.cc:460] Webserver started at http://127.0.98.1:39955/ using document root <none> and password file <none>
I20260926 09:32:56.096056 7232 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:56.096164 7232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:56.097874 7232 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:32:56.098637 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.098994 7232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:56.099118 7232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "8002dfb7e8bd4a9097332207c63fe0b2"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "12257c2a079ad2a8fa9857d163d4f525"
server_key_iv: "4eb9f6a14669d34e2192ffb9c6b7fb68"
server_key_version: "encryptionkey@0"
I20260926 09:32:56.099619 7232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:56.115783 7232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:56.118296 7232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:56.118466 7232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:56.118729 7232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:56.119078 7232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:56.119115 7232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.119139 7232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:56.119187 7232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.127135 7232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41441
I20260926 09:32:56.127173 7448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41441 every 8 connection(s)
I20260926 09:32:56.127642 7232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:32:56.130520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 7232
I20260926 09:32:56.130683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 6681
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:32:56.137068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35219
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=44325
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787,127.0.98.60:36145 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:32:56.141834 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.128395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55827 (local address 127.0.98.61:41787)
0926 09:32:56.128636 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.128639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.130090 (+ 1451us) server_negotiation.cc:407] Connection header received
0926 09:32:56.130118 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.130119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.130129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.130216 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.130910 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.131735 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.133374 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.133736 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.133776 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.137555 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.137556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.137557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.137565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.139248 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.140476 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.140477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.140478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.140557 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.140952 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.140953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.140953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.141122 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.141169 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.141617 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.141711 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260926 09:32:56.142307 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.128407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59655 (local address 127.0.98.62:39679)
0926 09:32:56.128457 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.128460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.130202 (+ 1742us) server_negotiation.cc:407] Connection header received
0926 09:32:56.130312 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.130314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.130328 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.130386 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.133065 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.133801 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.134707 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.134929 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.135007 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.138848 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.138850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.138850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.138858 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.141009 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.141451 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.141452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.141453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.141520 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.141719 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.141720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.141720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.141861 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.141908 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.142122 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.142203 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:32:56.142467 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.128391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.98.1:55827)
0926 09:32:56.130004 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.130014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.130066 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.130222 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.130224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.130519 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.130810 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.130817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.131867 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.131867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.133274 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.133276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.133851 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.133851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.133953 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.134885 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.134895 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.137452 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.140020 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.140028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.140030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.140380 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.140651 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.140652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.140653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.140857 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.141221 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.141228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.141412 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.141986 (+ 574us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.142208 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"mutex_wait_us":120,"thread_start_us":84,"threads_started":1}
I20260926 09:32:56.142469 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.128391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.98.1:44235)
0926 09:32:56.128641 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.128686 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.129654 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.130022 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.130028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.130517 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.130811 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.130817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.132159 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.132160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.133268 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.133270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.133804 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.133805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.133953 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.135094 (+ 1141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.135098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.137922 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.141258 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.141258 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.141259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.141484 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.141711 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.141712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.141712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.141767 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.142061 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.142062 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.142100 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.142148 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.142250 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":99,"thread_start_us":83,"threads_started":1}
I20260926 09:32:56.142674 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.128485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:59655)
0926 09:32:56.130150 (+ 1665us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.130158 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.130229 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.132732 (+ 2503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.132733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.132910 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.132961 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.132962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.133900 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.133900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.134610 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.134613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.135005 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.135006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.135068 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.135541 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.135546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.138740 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.141097 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.141098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.141099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.141364 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.141590 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.141591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.141591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.141654 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.141957 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.141958 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.142003 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.142402 (+ 399us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.142530 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1643,"thread_start_us":67,"threads_started":1}
I20260926 09:32:56.142988 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.128475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44235 (local address 127.0.98.60:36145)
0926 09:32:56.129794 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.129796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.129803 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:32:56.129833 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.129834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.129845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.129931 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.130917 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.132062 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.133370 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.133679 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.133718 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.138046 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.138047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.138047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.138055 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.141173 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.141576 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.141576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.141577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.141638 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.141834 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.141835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.141835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.141969 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.142711 (+ 742us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.142755 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.142850 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":57,"threads_started":1}
I20260926 09:32:56.143046 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41787
I20260926 09:32:56.143147 7451 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.143374 7451 heartbeater.cc:511] Master 127.0.98.61:41787 requested a full tablet report, sending...
I20260926 09:32:56.143482 7449 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36145
I20260926 09:32:56.143515 7449 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.143584 7449 heartbeater.cc:511] Master 127.0.98.60:36145 requested a full tablet report, sending...
I20260926 09:32:56.143723 7450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 09:32:56.143779 7450 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.143852 7450 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 09:32:56.143904 7085 ts_manager.cc:194] Registered new tserver with Master: 8002dfb7e8bd4a9097332207c63fe0b2 (127.0.98.1:41441)
I20260926 09:32:56.143918 7261 ts_manager.cc:194] Registered new tserver with Master: 8002dfb7e8bd4a9097332207c63fe0b2 (127.0.98.1:41441)
I20260926 09:32:56.144193 6485 ts_manager.cc:194] Re-registered known tserver with Master: 8002dfb7e8bd4a9097332207c63fe0b2 (127.0.98.1:41441)
I20260926 09:32:56.144635 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59655
W20260926 09:32:56.225924 7457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:56.226164 7457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:56.226190 7457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:56.227651 7457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:56.227727 7457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:56.227774 7457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:32:56.229439 7457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44325
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787,127.0.98.60:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:56.229724 7457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:56.229995 7457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:56.235495 7463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:56.235525 7465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:56.235532 7462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:56.235553 7457 server_base.cc:1034] running on GCE node
I20260926 09:32:56.235904 7457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:56.236125 7457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:56.237321 7457 hybrid_clock.cc:648] HybridClock initialized: now 1790415176237298 us; error 41 us; skew 500 ppm
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:56.239952 7457 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 09:32:56.240410 7457 webserver.cc:460] Webserver started at http://127.0.98.2:44325/ using document root <none> and password file <none>
I20260926 09:32:56.240651 7457 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:56.240738 7457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:56.242166 7457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:56.242792 7472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.242975 7457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:56.243088 7457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "ab31df190ad31a1fddb23a94111acd6e"
server_key_iv: "8da30c48cbd9a6790ff61be7a77a8f98"
server_key_version: "encryptionkey@0"
I20260926 09:32:56.243564 7457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:56.257784 7457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:56.260146 7457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:56.260330 7457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:56.260607 7457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:56.260932 7457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:56.260989 7457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.261024 7457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:56.261054 7457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.267206 7457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35219
I20260926 09:32:56.267275 7585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35219 every 8 connection(s)
I20260926 09:32:56.267709 7457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:32:56.273878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 7457
I20260926 09:32:56.274066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 6815
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:32:56.279847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40099
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=42343
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787,127.0.98.60:36145 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:32:56.280800 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.268656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43977 (local address 127.0.98.60:36145)
0926 09:32:56.268698 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.268700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.270197 (+ 1497us) server_negotiation.cc:407] Connection header received
0926 09:32:56.270226 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.270228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.270243 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.270313 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.271048 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.271860 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.273001 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.273321 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.273362 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.276740 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.276741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.276742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.276750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.278423 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.279048 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.279049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.279050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.279109 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.279770 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.279771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.279771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.279927 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.280001 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.280546 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.280653 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:56.281447 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.268547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.98.2:43977)
0926 09:32:56.270108 (+ 1561us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.270118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.270170 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.270360 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.270360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.270744 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.270934 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.270941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.271954 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.271955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.272915 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.272917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.273430 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.273431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.273486 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.274594 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.274600 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.276634 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.278513 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.278521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.278523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.278883 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.279184 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.279185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.279186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.279348 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.280178 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.280186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.280376 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.280503 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.281142 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"mutex_wait_us":47,"thread_start_us":110,"threads_started":1}
I20260926 09:32:56.282095 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.268604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38447 (local address 127.0.98.61:41787)
0926 09:32:56.268712 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.268714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.269009 (+ 295us) server_negotiation.cc:407] Connection header received
0926 09:32:56.270018 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.270020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.270033 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.270090 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.271054 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.271850 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.272968 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.273271 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.273315 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.279156 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.279158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.279158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.279184 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.280790 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.281156 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.281158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.281158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.281213 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.281426 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.281428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.281428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.281590 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.281665 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.281870 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.281967 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 09:32:56.282435 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.268545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.98.2:38447)
0926 09:32:56.268855 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.268903 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.269893 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.270226 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.270231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.270737 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.270934 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.270941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.271954 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.271955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.272862 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.272864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.273398 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.273399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.273465 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.273932 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.273937 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.279054 (+ 5117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.280863 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.280863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.280864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.281071 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.281290 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.281290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.281291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.281345 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.281696 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.281697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.281744 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.282176 (+ 432us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.282298 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":28,"thread_start_us":95,"threads_started":1}
I20260926 09:32:56.283149 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41787
I20260926 09:32:56.283252 7588 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.283486 7586 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36145
I20260926 09:32:56.283491 7588 heartbeater.cc:511] Master 127.0.98.61:41787 requested a full tablet report, sending...
I20260926 09:32:56.283588 7586 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.283741 7586 heartbeater.cc:511] Master 127.0.98.60:36145 requested a full tablet report, sending...
I20260926 09:32:56.284039 7085 ts_manager.cc:194] Registered new tserver with Master: ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219)
I20260926 09:32:56.284080 7261 ts_manager.cc:194] Registered new tserver with Master: ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219)
I20260926 09:32:56.285178 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.268596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33167 (local address 127.0.98.62:39679)
0926 09:32:56.268671 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.268674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.270306 (+ 1632us) server_negotiation.cc:407] Connection header received
0926 09:32:56.270368 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.270370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.270381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.270426 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.271032 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.271625 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.272651 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.272846 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.272960 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.281636 (+ 8676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.281638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.281639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.281647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.283792 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.284195 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.284197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.284197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.284260 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.284444 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.284445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.284445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.284610 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.284680 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.285047 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.285117 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:32:56.285655 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.268545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:33167)
0926 09:32:56.270244 (+ 1699us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.270251 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.270307 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.270439 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.270440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.270728 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.270933 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.270938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.271731 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.271732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.272540 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.272542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.272966 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.272968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.273099 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.273649 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.273659 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.280931 (+ 7272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.283876 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.283877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.283878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.284099 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.284333 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.284333 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.284333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.284392 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.284848 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.284849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.284906 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.285008 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.285145 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1669,"mutex_wait_us":175,"thread_start_us":69,"threads_started":1}
I20260926 09:32:56.286145 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 09:32:56.286214 7587 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.286291 7587 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 09:32:56.286569 6485 ts_manager.cc:194] Re-registered known tserver with Master: ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219)
I20260926 09:32:56.286952 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33167
W20260926 09:32:56.367210 7593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:32:56.367507 7593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:32:56.367535 7593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:32:56.368685 7593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:32:56.368729 7593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:32:56.368785 7593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:32:56.370244 7593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42343
--tserver_master_addrs=127.0.98.62:39679,127.0.98.61:41787,127.0.98.60:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:32:56.370491 7593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:32:56.370711 7593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:32:56.375912 7598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:56.375963 7599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:32:56.375912 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:32:56.376221 7593 server_base.cc:1034] running on GCE node
I20260926 09:32:56.376466 7593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:32:56.376713 7593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:32:56.377884 7593 hybrid_clock.cc:648] HybridClock initialized: now 1790415176377882 us; error 45 us; skew 500 ppm
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:32:56.380628 7593 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 09:32:56.381140 7593 webserver.cc:460] Webserver started at http://127.0.98.3:42343/ using document root <none> and password file <none>
I20260926 09:32:56.381377 7593 fs_manager.cc:359] Metadata directory not provided
I20260926 09:32:56.381462 7593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:32:56.382859 7593 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:56.383514 7608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.383683 7593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:32:56.383780 7593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "557dde2da1f84cb39c69b82507192090"
format_stamp: "Formatted at 2026-09-26 09:32:55 on dist-test-slave-21h8"
server_key: "d6627f52cc06f8f359f6c6239acd379c"
server_key_iv: "0e6c6d00f1f828852326523c0f549c24"
server_key_version: "encryptionkey@0"
I20260926 09:32:56.384171 7593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:32:56.400704 7593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:32:56.403057 7593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:32:56.403236 7593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:32:56.403527 7593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:32:56.403892 7593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:32:56.403947 7593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.403983 7593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:32:56.404011 7593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:56.410449 7593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40099
I20260926 09:32:56.410493 7721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40099 every 8 connection(s)
I20260926 09:32:56.410898 7593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:32:56.416806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 7593
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 09:32:56 dist-test-slave-21h8 krb5kdc[6440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415176, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:32:56.423658 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.412006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56721 (local address 127.0.98.60:36145)
0926 09:32:56.412086 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.412088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.412306 (+ 218us) server_negotiation.cc:407] Connection header received
0926 09:32:56.412874 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.412875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.412887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.412920 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.413819 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.414556 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.415772 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.416105 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.416199 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.420540 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.420541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.420542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.420550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.422320 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.422798 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.422800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.422800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.422858 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.423056 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.423057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.423057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.423188 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.423231 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.423467 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.423564 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:32:56.423826 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.411728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.98.3:37053)
0926 09:32:56.412208 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.412216 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.412757 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.413041 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.413043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.413387 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.413657 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.413663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.414498 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.414498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.415685 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.415687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.416138 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.416138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.416231 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.417121 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.417132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.419953 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.421976 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.421984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.421987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.422335 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.422596 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.422596 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.422597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.422751 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.423067 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.423074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.423252 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.423394 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.423576 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":389,"thread_start_us":97,"threads_started":1}
I20260926 09:32:56.423985 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.411728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.98.3:56721)
0926 09:32:56.412191 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.412203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.412757 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.413480 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.413481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.413660 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.413716 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.413718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.414638 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.414638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.415686 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.415687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.416198 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.416198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.416267 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.417217 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.417221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.420424 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.422418 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.422419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.422419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.422714 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.422930 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.422930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.422931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.422984 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.423278 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.423278 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.423340 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.423785 (+ 445us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.423888 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":78,"thread_start_us":90,"threads_started":1}
I20260926 09:32:56.424360 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.411863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37053 (local address 127.0.98.61:41787)
0926 09:32:56.411915 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.411918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.412288 (+ 370us) server_negotiation.cc:407] Connection header received
0926 09:32:56.412874 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.412875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.412887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.412920 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.413780 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.414377 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.415780 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.415985 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.416059 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.420045 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.420046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.420046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.420055 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.421872 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.422419 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.422420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.422421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.422496 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.422825 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.422827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.422827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.422965 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.423041 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.424170 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.424250 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:32:56.424677 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41787
I20260926 09:32:56.424790 7724 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.425031 7724 heartbeater.cc:511] Master 127.0.98.61:41787 requested a full tablet report, sending...
I20260926 09:32:56.425500 7722 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36145
I20260926 09:32:56.425568 7722 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.425642 7722 heartbeater.cc:511] Master 127.0.98.60:36145 requested a full tablet report, sending...
I20260926 09:32:56.425855 7085 ts_manager.cc:194] Registered new tserver with Master: 557dde2da1f84cb39c69b82507192090 (127.0.98.3:40099)
I20260926 09:32:56.425958 7261 ts_manager.cc:194] Registered new tserver with Master: 557dde2da1f84cb39c69b82507192090 (127.0.98.3:40099)
I20260926 09:32:56.429051 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.418585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.0.98.62:39679)
0926 09:32:56.419137 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.419139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.419146 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:32:56.419168 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.419169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.419181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.420088 (+ 907us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.420128 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.420834 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.422869 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.423180 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.423949 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.425378 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.425379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.425379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.425387 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.427562 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.428025 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.428026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.428026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.428086 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.428359 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.428360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.428360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.428506 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.428609 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.428788 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.428905 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":91,"threads_started":1}
I20260926 09:32:56.429952 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.418480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.0.98.61:41787)
0926 09:32:56.418598 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.418600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.418608 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:32:56.418631 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.418631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.418644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.419351 (+ 707us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.420933 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.421651 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.422866 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.423180 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.425251 (+ 2071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.425995 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.425996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.425997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.426003 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.428268 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.428650 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.428651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.428652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.428744 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.429055 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.429056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.429057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.429195 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.429695 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.429741 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.429841 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260926 09:32:56.430832 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.411728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:56733)
0926 09:32:56.411987 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.412066 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.412734 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.413019 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.413024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.413380 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:32:56.413661 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.413666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.414646 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.414647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.415592 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.415594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.416064 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.416065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.416231 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.417122 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:32:56.417132 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:32:56.426692 (+ 9560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:32:56.429137 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.429138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.429138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.429365 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.429604 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:32:56.429605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:32:56.429605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:32:56.429678 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:32:56.430494 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:32:56.430494 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:32:56.430543 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.430616 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.430732 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":21,"thread_start_us":81,"threads_started":1}
I20260926 09:32:56.430830 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.411730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56733 (local address 127.0.98.62:39679)
0926 09:32:56.412020 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.412023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.412174 (+ 151us) server_negotiation.cc:407] Connection header received
0926 09:32:56.412847 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.412849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.412862 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.412909 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.413757 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.414546 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.415684 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.415968 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.416035 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.426801 (+ 10766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.426803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.426803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.426811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.429043 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.429464 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.429465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.429465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.429524 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.429759 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.429760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.429760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.430180 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.430230 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.430645 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.430727 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260926 09:32:56.433421 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.418386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.0.98.60:36145)
0926 09:32:56.418945 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.418947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.418954 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:32:56.418976 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.418977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.418989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.419056 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.419750 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.426315 (+ 6565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.427541 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.427923 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.427993 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.430507 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.430509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.430509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.430516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.432247 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.432704 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.432705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.432705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.432766 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.432948 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.432949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.432949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.433082 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.433183 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.433247 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.433326 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":70,"threads_started":1}
I20260926 09:32:56.431170 6485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:56.433715 6485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:56.434577 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 09:32:56.434643 7723 heartbeater.cc:464] Registering TS with master...
I20260926 09:32:56.434731 7723 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 09:32:56.435005 6484 ts_manager.cc:194] Re-registered known tserver with Master: 557dde2da1f84cb39c69b82507192090 (127.0.98.3:40099)
I20260926 09:32:56.435524 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56733
I20260926 09:32:56.442164 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.438281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41441 (local address 127.0.0.1:50686)
0926 09:32:56.438635 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.438645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.438708 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.438959 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.438959 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.438961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:56.439018 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.439019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.440032 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.440033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.440947 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.440948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.441902 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.441903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.441977 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.441982 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.442032 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.442073 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":5,"thread_start_us":113,"threads_started":1}
I20260926 09:32:56.442186 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.438383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.0.98.1:41441)
0926 09:32:56.438578 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.438583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.438728 (+ 145us) server_negotiation.cc:407] Connection header received
0926 09:32:56.438782 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.438784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.438804 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.438892 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:56.439091 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.439941 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.441039 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.441805 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.441891 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.441981 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.442056 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.442100 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260926 09:32:56.443080 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.438231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40099 (local address 127.0.0.1:60128)
0926 09:32:56.438456 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.438469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.438512 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.438934 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.438935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.438937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:56.438986 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.438987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.440654 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.440655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.441790 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.441793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.442823 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.442824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.442888 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.442892 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.442956 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.443000 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":54}
I20260926 09:32:56.443086 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.438478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.0.98.3:40099)
0926 09:32:56.438736 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.438740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.438751 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:32:56.438798 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.438801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.438820 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.438933 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:56.439046 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.440540 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.441897 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.442712 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.442815 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.442902 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.442971 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.443010 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":80,"threads_started":1}
I20260926 09:32:56.443017 7383 tablet_service.cc:1467] Processing CreateTablet for tablet c816a712baf44fb9b3f357019e3ab26a (DEFAULT_TABLE table=test-table [id=f15ca90849f7401b8323685c9a7daf41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:56.443183 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.438423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.0.98.2:35219)
0926 09:32:56.438967 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.438971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.438981 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:56.439016 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.439017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.439034 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.439709 (+ 675us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:56.439770 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.440806 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.441722 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.442623 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.442706 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.442771 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.443045 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.443090 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":81,"threads_started":1}
I20260926 09:32:56.443005 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.438328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35219 (local address 127.0.0.1:41792)
0926 09:32:56.438795 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.438803 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.438852 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.439587 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.439588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.439591 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:56.439633 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.439635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.440876 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.440877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.441617 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.441619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.442731 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.442732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.442802 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.442807 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.442870 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.442923 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":88,"threads_started":1}
I20260926 09:32:56.443480 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c816a712baf44fb9b3f357019e3ab26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:56.446417 7748 tablet_bootstrap.cc:492] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: Bootstrap starting.
I20260926 09:32:56.447249 7748 tablet_bootstrap.cc:654] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:56.447176 7656 tablet_service.cc:1467] Processing CreateTablet for tablet c816a712baf44fb9b3f357019e3ab26a (DEFAULT_TABLE table=test-table [id=f15ca90849f7401b8323685c9a7daf41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:56.447273 7520 tablet_service.cc:1467] Processing CreateTablet for tablet c816a712baf44fb9b3f357019e3ab26a (DEFAULT_TABLE table=test-table [id=f15ca90849f7401b8323685c9a7daf41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:56.447623 7656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c816a712baf44fb9b3f357019e3ab26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:56.447623 7748 log.cc:824] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:56.447727 7520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c816a712baf44fb9b3f357019e3ab26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:56.448424 7748 tablet_bootstrap.cc:492] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: No bootstrap required, opened a new log
I20260926 09:32:56.448511 7748 ts_tablet_manager.cc:1397] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:32:56.450727 7750 tablet_bootstrap.cc:492] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: Bootstrap starting.
I20260926 09:32:56.451297 7750 tablet_bootstrap.cc:654] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:56.451717 7751 tablet_bootstrap.cc:492] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: Bootstrap starting.
I20260926 09:32:56.451601 7748 raft_consensus.cc:348] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.451720 7750 log.cc:824] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:56.451781 7748 raft_consensus.cc:374] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:56.451828 7748 raft_consensus.cc:729] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8002dfb7e8bd4a9097332207c63fe0b2, State: Initialized, Role: FOLLOWER
I20260926 09:32:56.451962 7748 consensus_queue.cc:260] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.452246 7450 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
I20260926 09:32:56.452471 7751 tablet_bootstrap.cc:654] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:56.452615 7748 ts_tablet_manager.cc:1428] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 09:32:56.452813 7751 log.cc:824] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: Log is configured to *not* fsync() on all Append() calls
I20260926 09:32:56.453661 7750 tablet_bootstrap.cc:492] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: No bootstrap required, opened a new log
I20260926 09:32:56.453734 7750 ts_tablet_manager.cc:1397] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:32:56.454531 7751 tablet_bootstrap.cc:492] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: No bootstrap required, opened a new log
I20260926 09:32:56.454589 7751 ts_tablet_manager.cc:1397] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:32:56.455834 7750 raft_consensus.cc:348] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.455989 7750 raft_consensus.cc:374] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:56.456034 7750 raft_consensus.cc:729] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec98c8347b9f46b4abb76f4c7a9b0e08, State: Initialized, Role: FOLLOWER
I20260926 09:32:56.456178 7750 consensus_queue.cc:260] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.456445 7750 ts_tablet_manager.cc:1428] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 09:32:56.456522 7587 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
I20260926 09:32:56.456713 7751 raft_consensus.cc:348] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.456871 7751 raft_consensus.cc:374] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:56.456903 7751 raft_consensus.cc:729] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557dde2da1f84cb39c69b82507192090, State: Initialized, Role: FOLLOWER
I20260926 09:32:56.457034 7751 consensus_queue.cc:260] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.457302 7751 ts_tablet_manager.cc:1428] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:32:56.457422 7723 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
I20260926 09:32:56.481293 7753 raft_consensus.cc:482] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:56.481416 7753 raft_consensus.cc:504] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.481748 7753 leader_election.cc:288] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 557dde2da1f84cb39c69b82507192090 (127.0.98.3:40099), ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219)
I20260926 09:32:56.484788 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.482139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57717 (local address 127.0.98.2:35219)
0926 09:32:56.482182 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.482184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.482193 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:56.482244 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.482245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.482257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.482331 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:56.482505 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.483000 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.484017 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.484396 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.484484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.484495 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.484665 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.484706 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:32:56.484790 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.482020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38745 (local address 127.0.98.3:40099)
0926 09:32:56.482071 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.482073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.482089 (+ 16us) server_negotiation.cc:407] Connection header received
0926 09:32:56.482155 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.482156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.482169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.482259 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:56.482422 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.482906 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.484022 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.484396 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.484491 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.484502 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.484680 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.484718 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:32:56.484797 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.482079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35219 (local address 127.0.98.1:57717)
0926 09:32:56.482132 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.482142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.482183 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.482334 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.482335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.482336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:56.482398 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.482399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.483101 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.483102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.483907 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.483909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.484485 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.484486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.484572 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.484576 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.484656 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.484703 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260926 09:32:56.484813 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.481934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40099 (local address 127.0.98.1:38745)
0926 09:32:56.482011 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.482027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.482095 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.482265 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.482266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.482268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:56.482342 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.482347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.483008 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.483009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.483905 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.483907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.484490 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.484491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.484587 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.484590 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.484670 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.484718 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":3}
I20260926 09:32:56.485374 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816a712baf44fb9b3f357019e3ab26a" candidate_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" is_pre_election: true
I20260926 09:32:56.485376 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816a712baf44fb9b3f357019e3ab26a" candidate_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557dde2da1f84cb39c69b82507192090" is_pre_election: true
I20260926 09:32:56.485550 7676 raft_consensus.cc:2454] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8002dfb7e8bd4a9097332207c63fe0b2 in term 0.
I20260926 09:32:56.485550 7540 raft_consensus.cc:2454] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8002dfb7e8bd4a9097332207c63fe0b2 in term 0.
I20260926 09:32:56.485796 7338 leader_election.cc:302] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8002dfb7e8bd4a9097332207c63fe0b2, ec98c8347b9f46b4abb76f4c7a9b0e08; no voters:
I20260926 09:32:56.485956 7753 raft_consensus.cc:2781] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:56.486052 7753 raft_consensus.cc:482] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:56.486086 7753 raft_consensus.cc:3037] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:56.486979 7753 raft_consensus.cc:504] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.487150 7753 leader_election.cc:288] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 557dde2da1f84cb39c69b82507192090 (127.0.98.3:40099), ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219)
I20260926 09:32:56.487445 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816a712baf44fb9b3f357019e3ab26a" candidate_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557dde2da1f84cb39c69b82507192090"
I20260926 09:32:56.487483 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c816a712baf44fb9b3f357019e3ab26a" candidate_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08"
I20260926 09:32:56.487555 7676 raft_consensus.cc:3037] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:56.487555 7540 raft_consensus.cc:3037] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:56.488489 7676 raft_consensus.cc:2454] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8002dfb7e8bd4a9097332207c63fe0b2 in term 1.
I20260926 09:32:56.488490 7540 raft_consensus.cc:2454] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8002dfb7e8bd4a9097332207c63fe0b2 in term 1.
I20260926 09:32:56.488647 7338 leader_election.cc:302] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 557dde2da1f84cb39c69b82507192090, 8002dfb7e8bd4a9097332207c63fe0b2; no voters:
I20260926 09:32:56.488766 7753 raft_consensus.cc:2781] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:56.488947 7753 raft_consensus.cc:686] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 1 LEADER]: Becoming Leader. State: Replica: 8002dfb7e8bd4a9097332207c63fe0b2, State: Running, Role: LEADER
I20260926 09:32:56.489049 7753 consensus_queue.cc:237] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:56.489938 6485 catalog_manager.cc:5495] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8002dfb7e8bd4a9097332207c63fe0b2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } health_report { overall_health: UNKNOWN } } }
I20260926 09:32:56.490873 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.487069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.0.1:43462)
0926 09:32:56.487166 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:56.487178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:56.487219 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:56.487570 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:56.487572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:56.487575 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:56.487624 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.487626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.488488 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.488489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.489609 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:56.489613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.490648 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.490652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:56.490703 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.490706 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:32:56.490739 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 09:32:56.490776 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:32:56.490883 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.487306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.0.98.60:36145)
0926 09:32:56.487381 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.487383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.487395 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:32:56.487428 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.487429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.487478 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.487563 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:56.487710 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.488397 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.489947 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.490545 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.490652 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.490730 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.490752 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.490802 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:32:56.491494 7280 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 09:32:56.492491 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf7ce0434d14c5283f588019c553a7a. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:56.492601 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:32:56.492683 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: true } } }
I20260926 09:32:56.492781 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:56.493584 7044 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: attempting to promote NON_VOTER 78b19be33668429f9b8e7fa432582542 to VOTER
I20260926 09:32:56.493752 7044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } }
I20260926 09:32:56.494258 7280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 LEARNER]: Refusing update from remote peer 5cf7ce0434d14c5283f588019c553a7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 09:32:56.494258 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Refusing update from remote peer 5cf7ce0434d14c5283f588019c553a7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 09:32:56.494482 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 09:32:56.494589 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 09:32:56.495275 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 78b19be33668429f9b8e7fa432582542 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:56.495440 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 78b19be33668429f9b8e7fa432582542 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:56.495551 7280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 78b19be33668429f9b8e7fa432582542 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:56.496311 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:56.496393 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:56.496403 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:56.496515 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:56.496531 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:56.496598 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:56.496608 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:56.496676 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:32:56.496824 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cf7ce0434d14c5283f588019c553a7a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:56.496901 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:32:56.506675 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.504364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.0.98.1:41441)
0926 09:32:56.504426 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.504428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.504437 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:56.504478 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.504478 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.504497 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.504573 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:56.504688 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.505172 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.505916 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.506134 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.506196 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.506375 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:56.506486 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:56.506550 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.506591 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.506614 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:32:56.508653 7540 raft_consensus.cc:1261] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 1 FOLLOWER]: Refusing update from remote peer 8002dfb7e8bd4a9097332207c63fe0b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:56.508659 7676 raft_consensus.cc:1261] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 1 FOLLOWER]: Refusing update from remote peer 8002dfb7e8bd4a9097332207c63fe0b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:56.509084 7757 consensus_queue.cc:1035] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:56.509214 7753 consensus_queue.cc:1035] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:56.515364 6484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:56.515496 6484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:56.517248 6484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cf7ce0434d14c5283f588019c553a7a: Sending DeleteTablet for 3 replicas of tablet c816a712baf44fb9b3f357019e3ab26a
I20260926 09:32:56.517642 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet c816a712baf44fb9b3f357019e3ab26a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50686
I20260926 09:32:56.517863 7520 tablet_service.cc:1514] Processing DeleteTablet for tablet c816a712baf44fb9b3f357019e3ab26a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41792
I20260926 09:32:56.517830 7656 tablet_service.cc:1514] Processing DeleteTablet for tablet c816a712baf44fb9b3f357019e3ab26a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60128
I20260926 09:32:56.518105 7778 tablet_replica.cc:330] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: stopping tablet replica
I20260926 09:32:56.518240 7777 tablet_replica.cc:330] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: stopping tablet replica
I20260926 09:32:56.518402 7777 raft_consensus.cc:2229] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:56.518617 7777 raft_consensus.cc:2258] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:56.518618 7778 raft_consensus.cc:2229] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:56.518718 7778 raft_consensus.cc:2258] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:56.518749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 6453
I20260926 09:32:56.520526 7777 ts_tablet_manager.cc:1905] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:56.522348 7777 ts_tablet_manager.cc:1918] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:56.522444 7777 log.cc:1196] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/c816a712baf44fb9b3f357019e3ab26a
I20260926 09:32:56.522681 7777 ts_tablet_manager.cc:1939] T c816a712baf44fb9b3f357019e3ab26a P 557dde2da1f84cb39c69b82507192090: Deleting consensus metadata
W20260926 09:32:56.525918 7610 connection.cc:535] server connection from 127.0.0.1:60128 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:32:56.527283 7778 ts_tablet_manager.cc:1905] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:56.529597 7778 ts_tablet_manager.cc:1918] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:56.529688 7778 log.cc:1196] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/c816a712baf44fb9b3f357019e3ab26a
I20260926 09:32:56.529982 7778 ts_tablet_manager.cc:1939] T c816a712baf44fb9b3f357019e3ab26a P ec98c8347b9f46b4abb76f4c7a9b0e08: Deleting consensus metadata
W20260926 09:32:56.530375 7474 connection.cc:441] server connection from 127.0.0.1:41792 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41792 (request call id 1) could send its response
I20260926 09:32:56.534247 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.518822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.0.98.60:36145)
0926 09:32:56.518863 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.518865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.523618 (+ 4753us) server_negotiation.cc:407] Connection header received
0926 09:32:56.523736 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.523738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.523749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.523842 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.524473 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.524978 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.528030 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.528354 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.528414 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.530337 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.530338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.530338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.530345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.532737 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.533224 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.533225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.533225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.533296 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.533525 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.533526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.533526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.533691 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.533800 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.534028 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.534110 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:32:56.536185 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.527813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.0.98.61:41787)
0926 09:32:56.527857 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.527859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.528028 (+ 169us) server_negotiation.cc:407] Connection header received
0926 09:32:56.528128 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.528129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.528144 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.528174 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.528941 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.529668 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.530699 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.531019 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.531062 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.532705 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.532706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.532707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.532714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.534857 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.535290 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.535291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.535291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.535381 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.535583 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.535584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.535584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.535713 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.535784 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.535987 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.536086 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:32:56.536281 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:56.527447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.0.98.60:36145)
0926 09:32:56.527492 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:56.527494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:56.527795 (+ 301us) server_negotiation.cc:407] Connection header received
0926 09:32:56.527977 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:56.527978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:56.527990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:56.528022 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:32:56.529160 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.529982 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.531209 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:56.531554 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:56.531605 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:56.533228 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:32:56.533229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:32:56.533229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:32:56.533235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:32:56.534780 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.535486 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.535487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.535487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.535541 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:32:56.535769 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:32:56.535771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:32:56.535771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:32:56.535885 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:32:56.535991 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:56.536088 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 09:32:56.536162 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:32:56.541014 7776 tablet_replica.cc:330] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: stopping tablet replica
I20260926 09:32:56.541179 7776 raft_consensus.cc:2229] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:56.541313 7776 raft_consensus.cc:2258] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:56.541823 7776 ts_tablet_manager.cc:1905] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:56.543990 7776 ts_tablet_manager.cc:1918] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:56.544090 7776 log.cc:1196] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/c816a712baf44fb9b3f357019e3ab26a
I20260926 09:32:56.544356 7776 ts_tablet_manager.cc:1939] T c816a712baf44fb9b3f357019e3ab26a P 8002dfb7e8bd4a9097332207c63fe0b2: Deleting consensus metadata
W20260926 09:32:56.544826 7338 connection.cc:441] server connection from 127.0.0.1:50686 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50686 (request call id 1) could send its response
I20260926 09:32:57.514014 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:57.513712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:53523)
0926 09:32:57.513832 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:57.513942 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":88,"threads_started":1}
W20260926 09:32:57.514542 7723 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:57.516842 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:57.516096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:42695)
0926 09:32:57.516255 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:57.516778 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
W20260926 09:32:57.517077 7587 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:57.518208 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:57.517984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:37663)
0926 09:32:57.518109 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:57.518170 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
W20260926 09:32:57.518411 7450 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:58.018148 7789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5cf7ce0434d14c5283f588019c553a7a)
I20260926 09:32:58.018162 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5cf7ce0434d14c5283f588019c553a7a)
I20260926 09:32:58.018285 7789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } }
I20260926 09:32:58.018322 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } }
I20260926 09:32:58.018682 7789 leader_election.cc:288] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679), 78b19be33668429f9b8e7fa432582542 (127.0.98.60:36145)
I20260926 09:32:58.018898 7790 leader_election.cc:288] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679), 3aba30e4d364422c91a0b571d7d3c8e6 (127.0.98.61:41787)
I20260926 09:32:58.019021 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.018812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37522)
0926 09:32:58.018922 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.018971 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
W20260926 09:32:58.019135 7257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:58.019650 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.019295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37524)
0926 09:32:58.019401 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.019463 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260926 09:32:58.019687 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.019602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37534)
0926 09:32:58.019634 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.019659 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 09:32:58.019766 7072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
W20260926 09:32:58.019881 7257 leader_election.cc:334] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:58.020285 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.020181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37550)
0926 09:32:58.020237 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.020262 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 09:32:58.020426 7072 leader_election.cc:334] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:58.022866 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.019385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.0.98.61:41787)
0926 09:32:58.019463 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:58.019465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:58.019474 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:32:58.019498 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:58.019499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:58.019510 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:58.019727 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:58.019754 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:58.020502 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:58.021938 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:58.022460 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:58.022563 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:58.022576 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:58.022757 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 09:32:58.022808 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63,"thread_start_us":38,"threads_started":1}
I20260926 09:32:58.022866 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.018828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41787 (local address 127.0.0.1:36834)
0926 09:32:58.018987 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.019007 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:58.019076 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:58.019616 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:58.019617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:58.019619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:58.019661 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:58.019663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:58.020578 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:58.020579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:58.021815 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:58.021818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:58.022565 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:58.022566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:58.022630 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:58.022634 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:58.022748 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:32:58.022796 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":39,"threads_started":1}
I20260926 09:32:58.022985 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.018893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36145 (local address 127.0.0.1:43488)
0926 09:32:58.019296 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.019336 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:58.019398 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:58.019581 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:58.019582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:58.019585 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:58.019644 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:58.019646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:58.020578 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:58.020579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:58.021815 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:58.021818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:58.022688 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:58.022689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:58.022768 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:58.022773 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:32:58.022827 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:32:58.022881 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":278,"threads_started":1}
I20260926 09:32:58.023154 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.019006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.98.60:36145)
0926 09:32:58.019117 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:58.019119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:58.019453 (+ 334us) server_negotiation.cc:407] Connection header received
0926 09:32:58.019482 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:58.019483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:58.019494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:58.019564 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:58.019724 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:58.020510 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:58.021916 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:58.022550 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:58.022636 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:58.022648 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:58.023046 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 09:32:58.023091 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":46,"threads_started":1}
I20260926 09:32:58.023205 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78b19be33668429f9b8e7fa432582542" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" is_pre_election: true
I20260926 09:32:58.023360 7095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b19be33668429f9b8e7fa432582542 in term 1.
I20260926 09:32:58.023415 7280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "78b19be33668429f9b8e7fa432582542" is_pre_election: true
I20260926 09:32:58.023525 7280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aba30e4d364422c91a0b571d7d3c8e6 in term 1.
I20260926 09:32:58.023722 7069 leader_election.cc:302] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aba30e4d364422c91a0b571d7d3c8e6, 78b19be33668429f9b8e7fa432582542; no voters: 5cf7ce0434d14c5283f588019c553a7a
I20260926 09:32:58.023783 7256 leader_election.cc:302] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aba30e4d364422c91a0b571d7d3c8e6, 78b19be33668429f9b8e7fa432582542; no voters: 5cf7ce0434d14c5283f588019c553a7a
I20260926 09:32:58.023874 7790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 09:32:58.023874 7789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 09:32:58.023906 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5cf7ce0434d14c5283f588019c553a7a)
I20260926 09:32:58.023916 7789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5cf7ce0434d14c5283f588019c553a7a)
I20260926 09:32:58.023934 7790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:32:58.023939 7789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:32:58.024649 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } }
I20260926 09:32:58.024693 7789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } }
I20260926 09:32:58.024794 7790 leader_election.cc:288] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 2 election: Requested vote from peers 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679), 3aba30e4d364422c91a0b571d7d3c8e6 (127.0.98.61:41787)
I20260926 09:32:58.024899 7789 leader_election.cc:288] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [CANDIDATE]: Term 2 election: Requested vote from peers 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679), 78b19be33668429f9b8e7fa432582542 (127.0.98.60:36145)
I20260926 09:32:58.025080 7280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "78b19be33668429f9b8e7fa432582542"
I20260926 09:32:58.025204 7280 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aba30e4d364422c91a0b571d7d3c8e6 in current term 2: Already voted for candidate 78b19be33668429f9b8e7fa432582542 in this term.
I20260926 09:32:58.025362 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.024902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37572)
0926 09:32:58.025274 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.025334 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
I20260926 09:32:58.025331 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78b19be33668429f9b8e7fa432582542" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3aba30e4d364422c91a0b571d7d3c8e6"
I20260926 09:32:58.025413 7095 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 78b19be33668429f9b8e7fa432582542 in current term 2: Already voted for candidate 3aba30e4d364422c91a0b571d7d3c8e6 in this term.
I20260926 09:32:58.025490 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.024952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37578)
0926 09:32:58.025422 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.025456 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451}
I20260926 09:32:58.025669 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.025587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37582)
0926 09:32:58.025621 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.025644 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 09:32:58.025733 7257 leader_election.cc:334] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:58.025766 7257 leader_election.cc:302] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78b19be33668429f9b8e7fa432582542; no voters: 3aba30e4d364422c91a0b571d7d3c8e6, 5cf7ce0434d14c5283f588019c553a7a
I20260926 09:32:58.025853 7790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 09:32:58.025930 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.025769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37586)
0926 09:32:58.025846 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.025872 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:32:58.026005 7072 leader_election.cc:334] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:58.026074 7072 leader_election.cc:302] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aba30e4d364422c91a0b571d7d3c8e6; no voters: 5cf7ce0434d14c5283f588019c553a7a, 78b19be33668429f9b8e7fa432582542
I20260926 09:32:58.026135 7789 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 09:32:58.515131 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.514880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:33001)
0926 09:32:58.515032 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.515077 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260926 09:32:58.517705 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.517496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:60377)
0926 09:32:58.517620 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.517655 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260926 09:32:58.518967 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:58.518750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:41541)
0926 09:32:58.518885 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:58.518919 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260926 09:32:59.515779 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.515535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:41035)
0926 09:32:59.515673 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.515721 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260926 09:32:59.518337 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.518134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:59489)
0926 09:32:59.518264 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.518299 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260926 09:32:59.519585 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.519395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:53595)
0926 09:32:59.519516 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.519550 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260926 09:32:59.636300 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:59.636410 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } }
I20260926 09:32:59.636619 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679), 3aba30e4d364422c91a0b571d7d3c8e6 (127.0.98.61:41787)
I20260926 09:32:59.636874 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78b19be33668429f9b8e7fa432582542" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" is_pre_election: true
I20260926 09:32:59.636960 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.636758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37810)
0926 09:32:59.636878 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.636921 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260926 09:32:59.636986 7095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b19be33668429f9b8e7fa432582542 in term 2.
I20260926 09:32:59.637132 7256 leader_election.cc:302] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aba30e4d364422c91a0b571d7d3c8e6, 78b19be33668429f9b8e7fa432582542; no voters:
I20260926 09:32:59.637238 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 09:32:59.637302 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:59.637324 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 2 FOLLOWER]: Advancing to term 3
I20260926 09:32:59.637401 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.637266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37816)
0926 09:32:59.637341 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.637377 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 09:32:59.637502 7257 leader_election.cc:334] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:59.638115 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } }
I20260926 09:32:59.638252 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 3 election: Requested vote from peers 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679), 3aba30e4d364422c91a0b571d7d3c8e6 (127.0.98.61:41787)
I20260926 09:32:59.638417 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.638323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37830)
0926 09:32:59.638367 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.638391 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 09:32:59.638441 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78b19be33668429f9b8e7fa432582542" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3aba30e4d364422c91a0b571d7d3c8e6"
I20260926 09:32:59.638540 7095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 2 FOLLOWER]: Advancing to term 3
I20260926 09:32:59.638712 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.638642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37834)
0926 09:32:59.638673 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.638695 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 09:32:59.638803 7257 leader_election.cc:334] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260926 09:32:59.639253 7095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b19be33668429f9b8e7fa432582542 in term 3.
I20260926 09:32:59.639432 7256 leader_election.cc:302] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aba30e4d364422c91a0b571d7d3c8e6, 78b19be33668429f9b8e7fa432582542; no voters: 5cf7ce0434d14c5283f588019c553a7a
I20260926 09:32:59.639539 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 09:32:59.639640 7805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [term 3 LEADER]: Becoming Leader. State: Replica: 78b19be33668429f9b8e7fa432582542, State: Running, Role: LEADER
I20260926 09:32:59.639730 7805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } }
I20260926 09:32:59.640085 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78b19be33668429f9b8e7fa432582542. Latest consensus state: current_term: 3 leader_uuid: "78b19be33668429f9b8e7fa432582542" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:59.640179 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:59.640376 7810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:32:59.640811 7810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:32:59.640902 7810 catalog_manager.cc:1259] Loaded cluster ID: 0408c85ef29c4ec2a29f310ecd0cd43d
I20260926 09:32:59.640934 7810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:32:59.641108 7810 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:32:59.641196 7810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Loaded TSK: 0
I20260926 09:32:59.641345 7810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:32:59.708868 7270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:32:59.710664 7270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:32:59.711748 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [term 3 FOLLOWER]: Refusing update from remote peer 78b19be33668429f9b8e7fa432582542: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 09:32:59.711793 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.711666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37864)
0926 09:32:59.711706 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.711739 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 09:32:59.711982 7808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 09:32:59.712165 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.712037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.0.1:37878)
0926 09:32:59.712086 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.712136 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:32:59.712337 7257 consensus_peers.cc:487] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 -> Peer 5cf7ce0434d14c5283f588019c553a7a (127.0.98.62:39679): Couldn't send request to peer 5cf7ce0434d14c5283f588019c553a7a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 09:32:59.712626 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78b19be33668429f9b8e7fa432582542. Latest consensus state: current_term: 3 leader_uuid: "78b19be33668429f9b8e7fa432582542" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:59.712725 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:32:59.713079 7811 mvcc.cc:205] Tried to move back new op lower bound from 7333540576096956416 to 7333540575805100032. Current Snapshot: MvccSnapshot[applied={T|T < 7333540576096956416}]
I20260926 09:32:59.713176 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "78b19be33668429f9b8e7fa432582542" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:59.713292 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542 [sys.catalog]: This master's current role is: LEADER
I20260926 09:32:59.713464 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "78b19be33668429f9b8e7fa432582542" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "5cf7ce0434d14c5283f588019c553a7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } peers { permanent_uuid: "3aba30e4d364422c91a0b571d7d3c8e6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41787 } attrs { promote: false } } peers { permanent_uuid: "78b19be33668429f9b8e7fa432582542" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36145 } attrs { promote: false } } }
I20260926 09:32:59.713544 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aba30e4d364422c91a0b571d7d3c8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:32:59.719070 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.715583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41441 (local address 127.0.0.1:50710)
0926 09:32:59.715608 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.715619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:59.715671 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:59.715994 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:59.715995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:59.715996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:59.716059 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.716061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.717051 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.717052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.718236 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.718238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.718810 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.718811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.718870 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.718874 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:59.718933 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:32:59.718981 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":43}
I20260926 09:32:59.719187 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.715650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40099 (local address 127.0.0.1:60132)
0926 09:32:59.715777 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.715787 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:59.715832 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:59.716348 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:59.716348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:59.716351 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:59.716389 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.716390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.717265 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.717265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.718170 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.718172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.718942 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.718942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.719010 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.719014 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:59.719043 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 09:32:59.719086 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":90,"threads_started":1}
I20260926 09:32:59.719434 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.715849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35219 (local address 127.0.0.1:41808)
0926 09:32:59.715948 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.715958 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:59.716002 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:59.716337 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:59.716338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:59.716340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:59.716382 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.716384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.717005 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.717006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.718236 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.718238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.718943 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.718943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.719006 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.719010 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:59.719059 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:32:59.719106 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":55,"thread_start_us":64,"threads_started":1}
I20260926 09:32:59.719416 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.715753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.0.98.3:40099)
0926 09:32:59.716206 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:59.716209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:59.716217 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:32:59.716241 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:59.716241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:59.716251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:59.716459 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:59.716496 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.717184 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.718250 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.718836 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.719244 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.719259 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:59.719277 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 09:32:59.719342 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":53,"threads_started":1}
I20260926 09:32:59.719599 7656 tablet_service.cc:1467] Processing CreateTablet for tablet b0c037d4431a47a9bc4f13781372556a (DEFAULT_TABLE table=test-table [id=4f531c147c934a6bbb9482f5a106653b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:59.719739 7656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c037d4431a47a9bc4f13781372556a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:59.719733 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.715659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.0.98.1:41441)
0926 09:32:59.715831 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:59.715833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:59.715841 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:32:59.715868 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:59.715868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:59.715880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:59.716156 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:59.716202 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.716988 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.718339 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.718721 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.718812 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.718825 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:59.719591 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 09:32:59.719644 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
I20260926 09:32:59.719936 7383 tablet_service.cc:1467] Processing CreateTablet for tablet b0c037d4431a47a9bc4f13781372556a (DEFAULT_TABLE table=test-table [id=4f531c147c934a6bbb9482f5a106653b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:59.720077 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c037d4431a47a9bc4f13781372556a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:59.720142 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.716054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.0.98.2:35219)
0926 09:32:59.716198 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:59.716200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:59.716208 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:32:59.716231 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:59.716232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:59.716248 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:59.716334 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:59.716436 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.716921 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.718309 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.718868 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.718953 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.718967 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:59.719064 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 09:32:59.719104 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260926 09:32:59.720335 7520 tablet_service.cc:1467] Processing CreateTablet for tablet b0c037d4431a47a9bc4f13781372556a (DEFAULT_TABLE table=test-table [id=4f531c147c934a6bbb9482f5a106653b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:32:59.720453 7520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c037d4431a47a9bc4f13781372556a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:32:59.721460 7822 tablet_bootstrap.cc:492] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: Bootstrap starting.
I20260926 09:32:59.721869 7822 tablet_bootstrap.cc:654] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:59.722060 7824 tablet_bootstrap.cc:492] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: Bootstrap starting.
I20260926 09:32:59.722143 7823 tablet_bootstrap.cc:492] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2: Bootstrap starting.
I20260926 09:32:59.722443 7824 tablet_bootstrap.cc:654] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:59.722491 7823 tablet_bootstrap.cc:654] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2: Neither blocks nor log segments found. Creating new log.
I20260926 09:32:59.722569 7822 tablet_bootstrap.cc:492] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: No bootstrap required, opened a new log
I20260926 09:32:59.722621 7822 ts_tablet_manager.cc:1397] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:59.722791 7822 raft_consensus.cc:348] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.722886 7822 raft_consensus.cc:374] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:59.722915 7822 raft_consensus.cc:729] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557dde2da1f84cb39c69b82507192090, State: Initialized, Role: FOLLOWER
I20260926 09:32:59.722972 7822 consensus_queue.cc:260] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.723052 7823 tablet_bootstrap.cc:492] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2: No bootstrap required, opened a new log
I20260926 09:32:59.723127 7823 ts_tablet_manager.cc:1397] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:59.723486 7824 tablet_bootstrap.cc:492] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: No bootstrap required, opened a new log
I20260926 09:32:59.723536 7824 ts_tablet_manager.cc:1397] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:59.723562 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.723444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:49271)
0926 09:32:59.723484 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.723513 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 09:32:59.723678 7824 raft_consensus.cc:348] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.723169 7822 ts_tablet_manager.cc:1428] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:32:59.723759 7824 raft_consensus.cc:374] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:59.723780 7824 raft_consensus.cc:729] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec98c8347b9f46b4abb76f4c7a9b0e08, State: Initialized, Role: FOLLOWER
I20260926 09:32:59.723810 7722 heartbeater.cc:503] Master 127.0.98.60:36145 was elected leader, sending a full tablet report...
I20260926 09:32:59.723830 7824 consensus_queue.cc:260] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.723815 7823 raft_consensus.cc:348] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.723896 7823 raft_consensus.cc:374] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:32:59.723939 7823 raft_consensus.cc:729] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8002dfb7e8bd4a9097332207c63fe0b2, State: Initialized, Role: FOLLOWER
I20260926 09:32:59.724011 7824 ts_tablet_manager.cc:1428] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:32:59.723991 7823 consensus_queue.cc:260] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.724164 7823 ts_tablet_manager.cc:1428] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:32:59.724390 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.724300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:53681)
0926 09:32:59.724339 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.724365 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 09:32:59.724494 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.724397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:52279)
0926 09:32:59.724438 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.724467 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 09:32:59.724854 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.724752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:39451)
0926 09:32:59.724793 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.724822 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 09:32:59.724850 7586 heartbeater.cc:503] Master 127.0.98.60:36145 was elected leader, sending a full tablet report...
I20260926 09:32:59.725129 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.724883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:35047)
0926 09:32:59.725068 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.725100 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260926 09:32:59.725297 7449 heartbeater.cc:503] Master 127.0.98.60:36145 was elected leader, sending a full tablet report...
I20260926 09:32:59.725541 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.725432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:40853)
0926 09:32:59.725482 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.725509 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:32:59.744524 7828 raft_consensus.cc:482] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:32:59.744663 7828 raft_consensus.cc:504] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.745036 7828 leader_election.cc:288] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8002dfb7e8bd4a9097332207c63fe0b2 (127.0.98.1:41441), ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219)
I20260926 09:32:59.748054 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.745343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36891 (local address 127.0.98.2:35219)
0926 09:32:59.745400 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:59.745402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:59.745416 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:32:59.745496 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:59.745497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:59.745509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:59.745574 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:59.745729 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.746253 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.747291 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.747697 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.747781 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.747794 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:59.747938 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 09:32:59.747978 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:32:59.748075 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.745152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41441 (local address 127.0.98.3:54113)
0926 09:32:59.745195 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.745207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:59.745267 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:59.745482 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:59.745483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:59.745486 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:59.745570 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.745572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.746259 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.746260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.747173 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.747176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.747807 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.747809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.747879 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.747883 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:59.747954 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:32:59.747991 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":31}
I20260926 09:32:59.748081 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.745230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54113 (local address 127.0.98.1:41441)
0926 09:32:59.745302 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:59.745305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:59.745315 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:32:59.745342 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:59.745343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:59.745355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:59.745409 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:32:59.745645 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.746190 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.747236 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.747696 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.747782 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.747793 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:59.747979 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 09:32:59.748018 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:32:59.748088 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.745196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35219 (local address 127.0.98.3:36891)
0926 09:32:59.745374 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.745383 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:32:59.745420 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:32:59.745586 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:32:59.745587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:32:59.745589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:32:59.745664 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.745666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.746356 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.746356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.747200 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:32:59.747202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.747782 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.747783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:32:59.747854 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.747858 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:32:59.747910 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:32:59.747968 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260926 09:32:59.748322 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c037d4431a47a9bc4f13781372556a" candidate_uuid: "557dde2da1f84cb39c69b82507192090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" is_pre_election: true
I20260926 09:32:59.748376 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c037d4431a47a9bc4f13781372556a" candidate_uuid: "557dde2da1f84cb39c69b82507192090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" is_pre_election: true
I20260926 09:32:59.748453 7540 raft_consensus.cc:2454] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557dde2da1f84cb39c69b82507192090 in term 0.
I20260926 09:32:59.748472 7403 raft_consensus.cc:2454] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557dde2da1f84cb39c69b82507192090 in term 0.
I20260926 09:32:59.748625 7611 leader_election.cc:302] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 557dde2da1f84cb39c69b82507192090, ec98c8347b9f46b4abb76f4c7a9b0e08; no voters:
I20260926 09:32:59.748775 7828 raft_consensus.cc:2781] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:32:59.748816 7828 raft_consensus.cc:482] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:32:59.748857 7828 raft_consensus.cc:3037] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:59.749603 7828 raft_consensus.cc:504] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.749751 7828 leader_election.cc:288] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [CANDIDATE]: Term 1 election: Requested vote from peers 8002dfb7e8bd4a9097332207c63fe0b2 (127.0.98.1:41441), ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219)
I20260926 09:32:59.749915 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c037d4431a47a9bc4f13781372556a" candidate_uuid: "557dde2da1f84cb39c69b82507192090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08"
I20260926 09:32:59.749979 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c037d4431a47a9bc4f13781372556a" candidate_uuid: "557dde2da1f84cb39c69b82507192090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8002dfb7e8bd4a9097332207c63fe0b2"
I20260926 09:32:59.750025 7540 raft_consensus.cc:3037] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:59.750036 7403 raft_consensus.cc:3037] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:32:59.750783 7403 raft_consensus.cc:2454] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557dde2da1f84cb39c69b82507192090 in term 1.
I20260926 09:32:59.750783 7540 raft_consensus.cc:2454] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557dde2da1f84cb39c69b82507192090 in term 1.
I20260926 09:32:59.750962 7611 leader_election.cc:302] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 557dde2da1f84cb39c69b82507192090, ec98c8347b9f46b4abb76f4c7a9b0e08; no voters:
I20260926 09:32:59.751065 7828 raft_consensus.cc:2781] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:32:59.751227 7828 raft_consensus.cc:686] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 1 LEADER]: Becoming Leader. State: Replica: 557dde2da1f84cb39c69b82507192090, State: Running, Role: LEADER
I20260926 09:32:59.751307 7828 consensus_queue.cc:237] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } }
I20260926 09:32:59.751638 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.751526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:34211)
0926 09:32:59.751560 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.751613 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 09:32:59.752061 7270 catalog_manager.cc:5495] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 reported cstate change: term changed from 0 to 1, leader changed from <none> to 557dde2da1f84cb39c69b82507192090 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "557dde2da1f84cb39c69b82507192090" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "557dde2da1f84cb39c69b82507192090" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 } health_report { overall_health: UNKNOWN } } }
I20260926 09:32:59.766021 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.763941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.0.98.3:40099)
0926 09:32:59.763990 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:32:59.763992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:32:59.763999 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:32:59.764055 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:32:59.764055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:32:59.764073 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:32:59.764133 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:32:59.764223 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.764673 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.765359 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:32:59.765544 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:32:59.765593 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:32:59.765718 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:32:59.765822 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:32:59.765879 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:32:59.765912 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 09:32:59.765931 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:32:59.767035 7540 raft_consensus.cc:1261] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 1 FOLLOWER]: Refusing update from remote peer 557dde2da1f84cb39c69b82507192090: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:59.767127 7403 raft_consensus.cc:1261] T b0c037d4431a47a9bc4f13781372556a P 8002dfb7e8bd4a9097332207c63fe0b2 [term 1 FOLLOWER]: Refusing update from remote peer 557dde2da1f84cb39c69b82507192090: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:32:59.767378 7832 consensus_queue.cc:1035] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec98c8347b9f46b4abb76f4c7a9b0e08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:59.767521 7828 consensus_queue.cc:1035] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8002dfb7e8bd4a9097332207c63fe0b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:32:59.768258 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.768150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:45359)
0926 09:32:59.768195 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.768224 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 09:32:59.768784 7835 mvcc.cc:205] Tried to move back new op lower bound from 7333540576323751936 to 7333540576262021120. Current Snapshot: MvccSnapshot[applied={T|T < 7333540576323751936}]
I20260926 09:32:59.769160 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.769066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:50757)
0926 09:32:59.769104 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.769129 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 09:32:59.769182 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.769077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:53937)
0926 09:32:59.769125 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.769152 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:32:59.769609 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.768944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:39075)
0926 09:32:59.769553 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.769584 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593}
I20260926 09:32:59.769639 7836 mvcc.cc:205] Tried to move back new op lower bound from 7333540576323751936 to 7333540576262021120. Current Snapshot: MvccSnapshot[applied={T|T < 7333540576323751936}]
I20260926 09:32:59.771302 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.770793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:53647)
0926 09:32:59.771239 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.771271 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430}
I20260926 09:32:59.771783 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.771377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:41859)
0926 09:32:59.771718 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.771752 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324}
I20260926 09:32:59.772147 7270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:59.772253 7270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:32:59.773527 7270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78b19be33668429f9b8e7fa432582542: Sending DeleteTablet for 3 replicas of tablet b0c037d4431a47a9bc4f13781372556a
I20260926 09:32:59.773878 7656 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c037d4431a47a9bc4f13781372556a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60132
I20260926 09:32:59.773892 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c037d4431a47a9bc4f13781372556a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50710
I20260926 09:32:59.774070 7520 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c037d4431a47a9bc4f13781372556a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:32:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41808
I20260926 09:32:59.774276 7847 tablet_replica.cc:330] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: stopping tablet replica
I20260926 09:32:59.774364 7847 raft_consensus.cc:2229] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:32:59.774464 7847 raft_consensus.cc:2258] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:59.774566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 7232
I20260926 09:32:59.774855 7847 ts_tablet_manager.cc:1905] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:59.776405 7845 tablet_replica.cc:330] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: stopping tablet replica
I20260926 09:32:59.776486 7845 raft_consensus.cc:2229] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:32:59.776572 7845 raft_consensus.cc:2258] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:32:59.776857 7845 ts_tablet_manager.cc:1905] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:32:59.777068 7847 ts_tablet_manager.cc:1918] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:59.777165 7847 log.cc:1196] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/b0c037d4431a47a9bc4f13781372556a
I20260926 09:32:59.777450 7847 ts_tablet_manager.cc:1939] T b0c037d4431a47a9bc4f13781372556a P ec98c8347b9f46b4abb76f4c7a9b0e08: Deleting consensus metadata
I20260926 09:32:59.777837 7256 catalog_manager.cc:4841] TS ec98c8347b9f46b4abb76f4c7a9b0e08 (127.0.98.2:35219): tablet b0c037d4431a47a9bc4f13781372556a (table test-table [id=4f531c147c934a6bbb9482f5a106653b]) successfully deleted
I20260926 09:32:59.778442 7845 ts_tablet_manager.cc:1918] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:32:59.778517 7845 log.cc:1196] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/b0c037d4431a47a9bc4f13781372556a
I20260926 09:32:59.778743 7845 ts_tablet_manager.cc:1939] T b0c037d4431a47a9bc4f13781372556a P 557dde2da1f84cb39c69b82507192090: Deleting consensus metadata
I20260926 09:32:59.779134 7256 catalog_manager.cc:4841] TS 557dde2da1f84cb39c69b82507192090 (127.0.98.3:40099): tablet b0c037d4431a47a9bc4f13781372556a (table test-table [id=4f531c147c934a6bbb9482f5a106653b]) successfully deleted
I20260926 09:32:59.787506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 7457
I20260926 09:32:59.787688 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:32:59.787547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41441 (local address 127.0.0.1:50724)
0926 09:32:59.787617 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:32:59.787647 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 09:32:59.787851 7255 catalog_manager.cc:4584] TS 8002dfb7e8bd4a9097332207c63fe0b2 (127.0.98.1:41441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0c037d4431a47a9bc4f13781372556a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41441: connect: Connection refused (error 111)
I20260926 09:32:59.792834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 7593
I20260926 09:32:59.798116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 6949
I20260926 09:32:59.802995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 7139
2026-09-26T09:32:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6202 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:32:59 dist-test-slave-21h8 krb5kdc[7850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:32:59 dist-test-slave-21h8 krb5kdc[7850](info): set up 2 sockets
Sep 26 09:32:59 dist-test-slave-21h8 krb5kdc[7850](info): commencing operation
krb5kdc: starting...
W20260926 09:33:00.855810 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:33:00 dist-test-slave-21h8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:33:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:33:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 09:33:00.974332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:01.058931 7863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:01.059213 7863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:01.059259 7863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:01.060614 7863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:33:01.060703 7863 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:33:01.060726 7863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:01.060750 7863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:01.060781 7863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:33:01.060808 7863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:33:01.062448 7863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41845
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:01.062757 7863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:01.063007 7863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:01.068042 7871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:01.068033 7869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:01.068038 7868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:01.068208 7863 server_base.cc:1034] running on GCE node
I20260926 09:33:01.068523 7863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:01.068827 7863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:01.069967 7863 hybrid_clock.cc:648] HybridClock initialized: now 1790415181069955 us; error 28 us; skew 500 ppm
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:01.072638 7863 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 09:33:01.073122 7863 webserver.cc:460] Webserver started at http://127.0.98.62:46193/ using document root <none> and password file <none>
I20260926 09:33:01.073392 7863 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:01.073480 7863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:01.073623 7863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:01.074712 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "d1910f3ab77f43e69590eb6a6f605489"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "6bb2b8dbdb3e687eb56c768b69a9b17d"
server_key_iv: "351850fec13c8d9d80687172c2255314"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.075076 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1910f3ab77f43e69590eb6a6f605489"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "6bb2b8dbdb3e687eb56c768b69a9b17d"
server_key_iv: "351850fec13c8d9d80687172c2255314"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.076514 7863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:33:01.077230 7878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.077396 7863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.077502 7863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "d1910f3ab77f43e69590eb6a6f605489"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "6bb2b8dbdb3e687eb56c768b69a9b17d"
server_key_iv: "351850fec13c8d9d80687172c2255314"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.077598 7863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:01.106573 7863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:01.109135 7863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:01.109313 7863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:01.114334 7863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41845
I20260926 09:33:01.114368 7940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41845 every 8 connection(s)
I20260926 09:33:01.114781 7863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:33:01.115415 7941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.117774 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: Bootstrap starting.
I20260926 09:33:01.118391 7941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.118700 7941 log.cc:824] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:01.119526 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: No bootstrap required, opened a new log
I20260926 09:33:01.119536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 7863
I20260926 09:33:01.119647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:33:01.119846 392 external_mini_cluster.cc:1428] Setting key 419892f1f11442549f465ca143839b57
I20260926 09:33:01.121986 7941 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1910f3ab77f43e69590eb6a6f605489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } }
I20260926 09:33:01.122161 7941 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.122224 7941 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1910f3ab77f43e69590eb6a6f605489, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.122392 7941 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1910f3ab77f43e69590eb6a6f605489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } }
I20260926 09:33:01.122462 7941 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:33:01.122484 7941 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:33:01.122537 7941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:01.123106 7941 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1910f3ab77f43e69590eb6a6f605489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } }
I20260926 09:33:01.123229 7941 leader_election.cc:302] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1910f3ab77f43e69590eb6a6f605489; no voters:
I20260926 09:33:01.123447 7941 leader_election.cc:288] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:33:01.123476 7946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:01.123589 7946 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [term 1 LEADER]: Becoming Leader. State: Replica: d1910f3ab77f43e69590eb6a6f605489, State: Running, Role: LEADER
I20260926 09:33:01.123679 7946 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1910f3ab77f43e69590eb6a6f605489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } }
I20260926 09:33:01.123737 7941 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:33:01.124025 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1910f3ab77f43e69590eb6a6f605489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1910f3ab77f43e69590eb6a6f605489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } } }
I20260926 09:33:01.124199 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:01.124384 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1910f3ab77f43e69590eb6a6f605489. Latest consensus state: current_term: 1 leader_uuid: "d1910f3ab77f43e69590eb6a6f605489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1910f3ab77f43e69590eb6a6f605489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } } }
I20260926 09:33:01.124578 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489 [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:01.124907 7953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:33:01.125525 7953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:33:01.127396 7953 catalog_manager.cc:1347] Generated new cluster ID: e2b6f28af13f401a816c4b5b6572f151
I20260926 09:33:01.127456 7953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:33:01.133074 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.120428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.0.98.62:41845)
0926 09:33:01.120640 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.120648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.120676 (+ 28us) server_negotiation.cc:407] Connection header received
0926 09:33:01.121965 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.121982 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.122034 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.122181 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.122965 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.124331 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.125523 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.126129 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.126252 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.129102 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.129106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.129106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.129118 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.131117 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.131566 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.131569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.131570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.131686 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.131859 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.131860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.131860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.132133 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.132275 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.132734 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.132868 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260926 09:33:01.142832 7953 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:33:01.143642 7953 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:33:01.153957 7953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: Generated new TSK 0
I20260926 09:33:01.154161 7953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 09:33:01.182677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:01.265630 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:01.265880 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:01.265920 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:01.267096 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:01.267170 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:01.267212 7967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:33:01.267251 7967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:33:01.268827 7967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:01.269145 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:01.269388 7967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:01.274797 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:01.274791 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:01.274917 7967 server_base.cc:1034] running on GCE node
W20260926 09:33:01.274828 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:01.275256 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:01.275556 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:01.276705 7967 hybrid_clock.cc:648] HybridClock initialized: now 1790415181276689 us; error 30 us; skew 500 ppm
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:01.279220 7967 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 09:33:01.279796 7967 webserver.cc:460] Webserver started at http://127.0.98.1:43319/ using document root <none> and password file <none>
I20260926 09:33:01.280047 7967 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:01.280094 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:01.280225 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:01.281353 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "15614bf72be54ebc8615d15ddad1ac47"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "16293857c0011fe1dc7eb3cbdb0e10eb"
server_key_iv: "0b9ac86006ee41be1e4c9a647da5d83a"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.281726 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "15614bf72be54ebc8615d15ddad1ac47"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "16293857c0011fe1dc7eb3cbdb0e10eb"
server_key_iv: "0b9ac86006ee41be1e4c9a647da5d83a"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.283124 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:33:01.283982 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.284221 7967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:33:01.284318 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "15614bf72be54ebc8615d15ddad1ac47"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "16293857c0011fe1dc7eb3cbdb0e10eb"
server_key_iv: "0b9ac86006ee41be1e4c9a647da5d83a"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.284394 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:01.314582 7967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:01.316659 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:01.316823 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:01.317448 7967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:01.317497 7967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.317520 7967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:01.317533 7967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:33:01.325284 7967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41247
I20260926 09:33:01.325733 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:33:01.325753 8097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41247 every 8 connection(s)
I20260926 09:33:01.327795 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.317917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41675 (local address 127.0.98.62:41845)
0926 09:33:01.317992 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.317994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.318197 (+ 203us) server_negotiation.cc:407] Connection header received
0926 09:33:01.318707 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.318708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.318726 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.318810 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.319671 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.320562 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.321655 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.322031 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.322120 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.324833 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.324835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.324835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.324844 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.326565 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.326919 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.326920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.326920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.326979 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.327191 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.327191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.327192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.327297 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.327424 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.327657 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.327728 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:33:01.327996 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.317885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.1:41675)
0926 09:33:01.318049 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.318090 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.318603 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.318924 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.318928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.319260 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:01.319565 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.319570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.320692 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.320693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.321553 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.321555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.322134 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.322135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.322300 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.322873 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:01.322885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:01.324677 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:01.326635 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.326639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.326640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.326859 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.327033 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.327033 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.327033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.327137 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.327416 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:01.327420 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:01.327537 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.327625 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.327748 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260926 09:33:01.328184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 7967
I20260926 09:33:01.328289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:33:01.328495 392 external_mini_cluster.cc:1428] Setting key 3c03127dea2b35cbf65499e1f1243ac1
I20260926 09:33:01.328552 8098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41845
I20260926 09:33:01.328625 8098 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:01.329370 8098 heartbeater.cc:511] Master 127.0.98.62:41845 requested a full tablet report, sending...
I20260926 09:33:01.329965 7894 ts_manager.cc:194] Registered new tserver with Master: 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247)
I20260926 09:33:01.330852 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41675
I20260926 09:33:01.336071 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.329591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45877 (local address 127.0.98.62:41845)
0926 09:33:01.329627 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.329629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.329758 (+ 129us) server_negotiation.cc:407] Connection header received
0926 09:33:01.329812 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.329813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.329822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.329871 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.330258 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.330965 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.332170 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.332354 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.332413 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.333458 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.333459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.333459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.333465 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.334865 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.335253 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.335254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.335254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.335301 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.335549 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.335550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.335550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.335663 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.335734 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.335898 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.335987 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:33:01.336081 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.329551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.1:45877)
0926 09:33:01.329705 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.329715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.329770 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.329876 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.329877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.330116 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:01.330181 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.330183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.331073 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.331074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.332083 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.332086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.332435 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.332436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.332521 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.332929 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:01.332935 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:01.333378 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:01.334952 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.334953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.334953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.335177 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.335399 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.335400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.335400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.335481 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.335737 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:01.335738 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:01.335782 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.335860 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.335971 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 09:33:01.357712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:01.440385 8107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:01.440630 8107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:01.440670 8107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:01.441849 8107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:01.441919 8107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:01.441956 8107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:33:01.442003 8107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:33:01.443562 8107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:01.443835 8107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:01.444087 8107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:01.449492 8112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:01.449503 8115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:01.449683 8107 server_base.cc:1034] running on GCE node
W20260926 09:33:01.449507 8113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:01.449983 8107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:01.450242 8107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:01.451388 8107 hybrid_clock.cc:648] HybridClock initialized: now 1790415181451366 us; error 30 us; skew 500 ppm
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:01.453902 8107 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 09:33:01.454308 8107 webserver.cc:460] Webserver started at http://127.0.98.2:45463/ using document root <none> and password file <none>
I20260926 09:33:01.454497 8107 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:01.454538 8107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:01.454629 8107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:01.455755 8107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "055c9c7afaf44ddf9433d7ade3abafea"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "c017975084e1ec34fffe9e39aa95d2ac"
server_key_iv: "618478d06d34a528bb0374fe56703ac3"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.456063 8107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "055c9c7afaf44ddf9433d7ade3abafea"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "c017975084e1ec34fffe9e39aa95d2ac"
server_key_iv: "618478d06d34a528bb0374fe56703ac3"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.457348 8107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:33:01.458041 8122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.458206 8107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:33:01.458290 8107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "055c9c7afaf44ddf9433d7ade3abafea"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "c017975084e1ec34fffe9e39aa95d2ac"
server_key_iv: "618478d06d34a528bb0374fe56703ac3"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.458348 8107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:01.472896 8107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:01.474731 8107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:01.474841 8107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:01.475370 8107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:01.475418 8107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.475461 8107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:01.475495 8107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.482703 8107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32841
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:33:01.482769 8237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32841 every 8 connection(s)
I20260926 09:33:01.483197 8107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:33:01.485723 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.475805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34189 (local address 127.0.98.62:41845)
0926 09:33:01.475855 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.475857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.476115 (+ 258us) server_negotiation.cc:407] Connection header received
0926 09:33:01.476638 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.476640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.476653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.476740 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.477574 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.478414 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.480143 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.480477 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.480556 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.483293 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.483295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.483295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.483303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.484596 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.485002 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.485003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.485003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.485053 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.485251 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.485251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.485252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.485350 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.485397 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.485601 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.485660 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:33:01.485942 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.475803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.2:34189)
0926 09:33:01.475979 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.476021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.476540 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.476792 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.476795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.477162 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:01.477467 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.477473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.478531 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.478532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.480041 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.480045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.480596 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.480598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.480743 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.481320 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:01.481329 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:01.483189 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:01.484683 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.484688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.484689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.484928 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.485110 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.485110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.485111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.485200 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.485387 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:01.485390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:01.485499 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.485575 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.485693 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":120,"threads_started":1}
I20260926 09:33:01.486413 8238 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41845
I20260926 09:33:01.486519 8238 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:01.486763 8238 heartbeater.cc:511] Master 127.0.98.62:41845 requested a full tablet report, sending...
I20260926 09:33:01.487179 7895 ts_manager.cc:194] Registered new tserver with Master: 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:01.487681 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34189
I20260926 09:33:01.492791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 8107
I20260926 09:33:01.492821 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.487503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43955 (local address 127.0.98.62:41845)
0926 09:33:01.487544 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.487546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.487662 (+ 116us) server_negotiation.cc:407] Connection header received
0926 09:33:01.487750 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.487751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.487761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.487811 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.488142 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.488706 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.489398 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.489606 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.489683 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.490487 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.490489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.490489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.490496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.491654 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.492015 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.492017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.492017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.492074 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.492273 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.492274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.492274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.492415 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.492495 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.492650 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.492723 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:33:01.492836 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.487452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.2:43955)
0926 09:33:01.487588 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.487598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.487676 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.487815 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.487816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.488006 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:01.488056 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.488057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.488799 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.488800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.489314 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.489315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.489682 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.489683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.489747 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.490119 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:01.490123 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:01.490388 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:01.491749 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.491750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.491751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.491920 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.492140 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.492141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.492142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.492199 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.492489 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:01.492490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:01.492534 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.492626 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.492725 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260926 09:33:01.492939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:33:01.493158 392 external_mini_cluster.cc:1428] Setting key ea3dbd7aaecbc61ed5d4b41380bff886
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 09:33:01.520431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:01.602417 8247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:01.602628 8247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:01.602653 8247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:01.603924 8247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:01.603972 8247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:01.604014 8247 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:33:01.604051 8247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:33:01.605522 8247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:01.605778 8247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:01.606015 8247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:33:01.611105 8247 server_base.cc:1034] running on GCE node
W20260926 09:33:01.611145 8253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:01.611150 8252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:01.611207 8255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:01.611503 8247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:01.611753 8247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:01.612896 8247 hybrid_clock.cc:648] HybridClock initialized: now 1790415181612875 us; error 37 us; skew 500 ppm
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:01.615638 8247 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 09:33:01.616144 8247 webserver.cc:460] Webserver started at http://127.0.98.3:40845/ using document root <none> and password file <none>
I20260926 09:33:01.616380 8247 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:01.616427 8247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:01.616552 8247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:01.617746 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "731039aab732446994aa11f3e7a9b226"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "9f294eff5f303d250f82763cd58432f2"
server_key_iv: "aa039d336894a65e7335220416ffcda4"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.618089 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "731039aab732446994aa11f3e7a9b226"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "9f294eff5f303d250f82763cd58432f2"
server_key_iv: "aa039d336894a65e7335220416ffcda4"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.619624 8247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 09:33:01.620357 8262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.620549 8247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:01.620653 8247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "731039aab732446994aa11f3e7a9b226"
format_stamp: "Formatted at 2026-09-26 09:33:01 on dist-test-slave-21h8"
server_key: "9f294eff5f303d250f82763cd58432f2"
server_key_iv: "aa039d336894a65e7335220416ffcda4"
server_key_version: "encryptionkey@0"
I20260926 09:33:01.620726 8247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:01.630904 8247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:01.633530 8247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:01.633709 8247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:01.634297 8247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:01.634369 8247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.634423 8247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:01.634450 8247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.642416 8247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40255
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:33:01.642491 8377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40255 every 8 connection(s)
I20260926 09:33:01.642848 8247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:33:01.644897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 8247
I20260926 09:33:01.644977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:33:01.645169 392 external_mini_cluster.cc:1428] Setting key b50364d5751a170f25a85c16ffae18d8
I20260926 09:33:01.646310 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.634876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39263 (local address 127.0.98.62:41845)
0926 09:33:01.634966 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.634969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.635177 (+ 208us) server_negotiation.cc:407] Connection header received
0926 09:33:01.635743 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.635744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.635757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.635846 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.637043 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.637988 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.639694 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.640030 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.640142 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.643014 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.643015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.643016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.643023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.644634 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.645260 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.645261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.645261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.645333 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.645618 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.645619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.645619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.645777 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.645864 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.646151 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.646245 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 09:33:01.646430 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.634837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.3:39263)
0926 09:33:01.635034 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.635076 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.635636 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.635926 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.635929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.636228 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:01.636751 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.636760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.638110 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.638111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.639591 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.639595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.640190 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.640191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.640346 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.640989 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:01.641002 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:01.642866 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:01.644826 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.644833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.644835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.645157 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.645390 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.645391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.645391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.645539 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.645835 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:01.645842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:01.646014 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.646104 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.646238 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260926 09:33:01.646934 8378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41845
I20260926 09:33:01.647020 8378 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:01.647205 8378 heartbeater.cc:511] Master 127.0.98.62:41845 requested a full tablet report, sending...
I20260926 09:33:01.647651 7895 ts_manager.cc:194] Registered new tserver with Master: 731039aab732446994aa11f3e7a9b226 (127.0.98.3:40255)
I20260926 09:33:01.648173 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39263
I20260926 09:33:01.653234 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.648153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51413 (local address 127.0.98.62:41845)
0926 09:33:01.648191 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.648193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.648218 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:33:01.648296 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.648297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.648307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.648358 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.648715 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.649296 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.649948 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.650121 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.650173 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.651171 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.651172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.651173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.651179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.652361 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.652655 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.652656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.652656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.652702 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.652846 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.652847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.652847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.652954 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.653012 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.653124 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.653175 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:33:01.653251 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.648043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.3:51413)
0926 09:33:01.648165 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.648175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.648232 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.648375 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.648376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.648584 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:01.648639 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.648641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.649391 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.649392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.649878 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.649879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.650185 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.650186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.650230 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.650691 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:01.650697 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:01.651057 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:01.652434 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.652434 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.652435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.652590 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.652741 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.652741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.652742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.652782 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.653024 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:01.653025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:01.653057 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.653110 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.653182 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260926 09:33:01.657648 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:33:01.679215 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:01.673772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.0.98.62:41845)
0926 09:33:01.673840 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.673846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.673853 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:33:01.673876 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.673876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.673885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.673946 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.674523 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.675038 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.675878 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.676078 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.676162 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.679097 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34658: BlockingRecv error: recv got EOF from 127.0.0.1:34658 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:33:01.686151 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.679985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.0.98.62:41845)
0926 09:33:01.680052 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.680054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.680108 (+ 54us) server_negotiation.cc:407] Connection header received
0926 09:33:01.680144 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.680145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.680153 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.680201 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.680662 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.681133 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.681753 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.681931 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.681978 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.684009 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.684010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.684010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.684017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.685237 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.685540 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.685540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.685541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.685595 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.685755 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.685755 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.685756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.685858 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.685931 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.686030 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.686086 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:33:01.687808 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:33:01.689388 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:33:01.695978 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.691733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32841 (local address 127.0.0.1:37318)
0926 09:33:01.692154 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.692164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.692217 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.692564 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.692564 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.692567 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:01.692619 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.692620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.693508 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.693510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.694714 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.694716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.695673 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.695674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.695757 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.695772 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.695849 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.695886 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260926 09:33:01.696027 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.692184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.0.98.2:32841)
0926 09:33:01.692377 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.692381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.692390 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:33:01.692441 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.692444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.692460 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.692694 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:01.692749 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.693414 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.694803 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.695587 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.695662 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.695735 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.695876 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.695915 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":118,"threads_started":1}
I20260926 09:33:01.696214 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.691975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.0.98.3:40255)
0926 09:33:01.692138 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.692143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.692154 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:33:01.692197 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.692199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.692215 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.692297 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:01.692567 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.693765 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.695040 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.695623 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.695944 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.696030 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.696057 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.696116 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260926 09:33:01.696481 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.691976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.0.98.1:41247)
0926 09:33:01.692424 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.692429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.692438 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:33:01.692492 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.692494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.692510 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.692586 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:01.692777 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.693830 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.695076 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.695810 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.695895 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.695978 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.696353 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.696393 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":87,"threads_started":1}
I20260926 09:33:01.696278 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.691723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41247 (local address 127.0.0.1:46670)
0926 09:33:01.692092 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.692101 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.692137 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.692663 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.692663 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.692666 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:01.692701 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.692702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.693914 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.693915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.694986 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.694988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.696015 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.696016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.696088 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.696093 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.696147 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.696188 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":30,"thread_start_us":58,"threads_started":1}
I20260926 09:33:01.697165 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 0059ef52a79e4654a540949d2b50a3f0 (DEFAULT_TABLE table=test-table [id=0856e5c800224f9cad8ee743a2b9ccce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:01.697487 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0059ef52a79e4654a540949d2b50a3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.696617 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.691720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40255 (local address 127.0.0.1:46296)
0926 09:33:01.691874 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.691913 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.691987 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.692321 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.692325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.692327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:01.692462 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.692465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.693878 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.693879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.694945 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.694947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.695739 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.695740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.695796 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.695800 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.695830 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.695886 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":43,"thread_start_us":99,"threads_started":1}
I20260926 09:33:01.700043 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 0059ef52a79e4654a540949d2b50a3f0 (DEFAULT_TABLE table=test-table [id=0856e5c800224f9cad8ee743a2b9ccce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:01.700256 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 0059ef52a79e4654a540949d2b50a3f0 (DEFAULT_TABLE table=test-table [id=0856e5c800224f9cad8ee743a2b9ccce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:01.700392 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0059ef52a79e4654a540949d2b50a3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.700551 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0059ef52a79e4654a540949d2b50a3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.700629 8406 tablet_bootstrap.cc:492] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Bootstrap starting.
I20260926 09:33:01.701195 8406 tablet_bootstrap.cc:654] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.701429 8406 log.cc:824] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:01.702096 8406 tablet_bootstrap.cc:492] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: No bootstrap required, opened a new log
I20260926 09:33:01.702181 8406 ts_tablet_manager.cc:1397] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:33:01.703824 8406 raft_consensus.cc:348] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.703989 8406 raft_consensus.cc:374] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.704033 8406 raft_consensus.cc:729] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055c9c7afaf44ddf9433d7ade3abafea, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.704139 8406 consensus_queue.cc:260] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.704419 8406 ts_tablet_manager.cc:1428] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:33:01.704470 8408 tablet_bootstrap.cc:492] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: Bootstrap starting.
I20260926 09:33:01.704514 8409 tablet_bootstrap.cc:492] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: Bootstrap starting.
I20260926 09:33:01.704546 8238 heartbeater.cc:503] Master 127.0.98.62:41845 was elected leader, sending a full tablet report...
I20260926 09:33:01.705116 8409 tablet_bootstrap.cc:654] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.705116 8408 tablet_bootstrap.cc:654] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.705446 8409 log.cc:824] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:01.705446 8408 log.cc:824] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:01.706233 8408 tablet_bootstrap.cc:492] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: No bootstrap required, opened a new log
I20260926 09:33:01.706233 8409 tablet_bootstrap.cc:492] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: No bootstrap required, opened a new log
I20260926 09:33:01.706307 8409 ts_tablet_manager.cc:1397] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:33:01.706307 8408 ts_tablet_manager.cc:1397] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:33:01.708340 8408 raft_consensus.cc:348] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.708340 8409 raft_consensus.cc:348] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.708505 8409 raft_consensus.cc:374] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.708505 8408 raft_consensus.cc:374] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.708541 8408 raft_consensus.cc:729] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.708541 8409 raft_consensus.cc:729] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15614bf72be54ebc8615d15ddad1ac47, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.708660 8409 consensus_queue.cc:260] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.708660 8408 consensus_queue.cc:260] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.708932 8409 ts_tablet_manager.cc:1428] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:33:01.708927 8408 ts_tablet_manager.cc:1428] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:33:01.708956 8378 heartbeater.cc:503] Master 127.0.98.62:41845 was elected leader, sending a full tablet report...
I20260926 09:33:01.708956 8098 heartbeater.cc:503] Master 127.0.98.62:41845 was elected leader, sending a full tablet report...
I20260926 09:33:01.724051 8413 raft_consensus.cc:482] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:33:01.724205 8413 raft_consensus.cc:504] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.724689 8413 leader_election.cc:288] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247), 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:01.727962 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.724981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38783 (local address 127.0.98.1:41247)
0926 09:33:01.725047 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.725049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.725256 (+ 207us) server_negotiation.cc:407] Connection header received
0926 09:33:01.725360 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.725361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.725375 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.725436 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:01.725580 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.726108 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.727126 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.727601 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.727681 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.727692 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.727843 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.727886 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:33:01.728039 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.724979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44581 (local address 127.0.98.2:32841)
0926 09:33:01.725052 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.725054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.725067 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:33:01.725104 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.725105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.725118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.725191 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:01.725384 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.726008 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.727182 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.727634 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.727732 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.727743 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.727917 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.727959 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:33:01.727990 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.724884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41247 (local address 127.0.98.3:38783)
0926 09:33:01.725202 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.725212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.725259 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.725445 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.725446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.725447 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:01.725505 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.725506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.726194 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.726194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.727047 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.727049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.727680 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.727680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.727760 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.727763 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.727825 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.727897 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":126,"thread_start_us":114,"threads_started":1}
I20260926 09:33:01.728371 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.724883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32841 (local address 127.0.98.3:44581)
0926 09:33:01.724970 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.724988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.725052 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.725222 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.725223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.725225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:01.725297 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.725299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.726103 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.726105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.727052 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.727055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.727731 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.727732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.727812 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.727815 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.727888 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.727951 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 09:33:01.728664 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059ef52a79e4654a540949d2b50a3f0" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15614bf72be54ebc8615d15ddad1ac47" is_pre_election: true
I20260926 09:33:01.728732 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059ef52a79e4654a540949d2b50a3f0" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea" is_pre_election: true
I20260926 09:33:01.728858 8052 raft_consensus.cc:2454] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 0.
I20260926 09:33:01.728914 8192 raft_consensus.cc:2454] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 0.
I20260926 09:33:01.729138 8265 leader_election.cc:302] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15614bf72be54ebc8615d15ddad1ac47, 731039aab732446994aa11f3e7a9b226; no voters:
I20260926 09:33:01.729326 8413 raft_consensus.cc:2781] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:33:01.729386 8413 raft_consensus.cc:482] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:33:01.729425 8413 raft_consensus.cc:3037] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:01.730396 8413 raft_consensus.cc:504] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.730572 8413 leader_election.cc:288] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 election: Requested vote from peers 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247), 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:01.730718 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059ef52a79e4654a540949d2b50a3f0" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15614bf72be54ebc8615d15ddad1ac47"
I20260926 09:33:01.730726 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059ef52a79e4654a540949d2b50a3f0" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea"
I20260926 09:33:01.730805 8192 raft_consensus.cc:3037] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:01.730805 8052 raft_consensus.cc:3037] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:01.731806 8192 raft_consensus.cc:2454] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 1.
I20260926 09:33:01.731806 8052 raft_consensus.cc:2454] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 1.
I20260926 09:33:01.732007 8265 leader_election.cc:302] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15614bf72be54ebc8615d15ddad1ac47, 731039aab732446994aa11f3e7a9b226; no voters:
I20260926 09:33:01.732115 8413 raft_consensus.cc:2781] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:01.732316 8413 raft_consensus.cc:686] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 1 LEADER]: Becoming Leader. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Running, Role: LEADER
I20260926 09:33:01.732426 8413 consensus_queue.cc:237] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } }
I20260926 09:33:01.733330 7895 catalog_manager.cc:5495] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731039aab732446994aa11f3e7a9b226 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } health_report { overall_health: UNKNOWN } } }
W20260926 09:33:01.734179 8239 tablet.cc:2369] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:33:01.740826 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.738598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.0.98.3:40255)
0926 09:33:01.738635 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.738637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.738645 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:33:01.738716 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.738717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.738733 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.738786 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:33:01.738909 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.739423 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.740127 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.740297 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.740373 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.740575 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:33:01.740662 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:33:01.740725 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.740758 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.740775 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:33:01.742493 8192 raft_consensus.cc:1261] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Refusing update from remote peer 731039aab732446994aa11f3e7a9b226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:01.742493 8052 raft_consensus.cc:1261] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Refusing update from remote peer 731039aab732446994aa11f3e7a9b226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:01.742818 8413 consensus_queue.cc:1035] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:01.742964 8416 consensus_queue.cc:1035] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:01.745931 8420 mvcc.cc:205] Tried to move back new op lower bound from 7333540584414773248 to 7333540584376836096. Current Snapshot: MvccSnapshot[applied={T|T < 7333540584414773248 or (T in {7333540584414773248})}]
I20260926 09:33:01.746110 8419 mvcc.cc:205] Tried to move back new op lower bound from 7333540584414773248 to 7333540584376836096. Current Snapshot: MvccSnapshot[applied={T|T < 7333540584414773248 or (T in {7333540584414773248})}]
I20260926 09:33:01.747485 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:01.747571 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:01.748323 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: Sending DeleteTablet for 3 replicas of tablet 0059ef52a79e4654a540949d2b50a3f0
I20260926 09:33:01.748625 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0059ef52a79e4654a540949d2b50a3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46670
I20260926 09:33:01.748821 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet 0059ef52a79e4654a540949d2b50a3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37318
I20260926 09:33:01.748860 8429 tablet_replica.cc:330] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: stopping tablet replica
I20260926 09:33:01.748948 8429 raft_consensus.cc:2229] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:01.749056 8429 raft_consensus.cc:2258] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:01.749003 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 09:33:01.749080 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 0059ef52a79e4654a540949d2b50a3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46296
W20260926 09:33:01.749147 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:33:01.749310 8431 tablet_replica.cc:330] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: stopping tablet replica
I20260926 09:33:01.749408 8431 raft_consensus.cc:2229] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:33:01.749496 8431 raft_consensus.cc:2258] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:01.749706 8429 ts_tablet_manager.cc:1905] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:01.749902 8431 ts_tablet_manager.cc:1905] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:01.750834 8309 tablet_service.cc:1467] Processing CreateTablet for tablet a1dd2c67e4c5489c99e8ebe8302bdd60 (DEFAULT_TABLE table=test-table [id=4e12ceba2e754c65b69a6f8fc0fb3c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:01.750955 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1dd2c67e4c5489c99e8ebe8302bdd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.751636 8431 ts_tablet_manager.cc:1918] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:33:01.751700 8431 log.cc:1196] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/0059ef52a79e4654a540949d2b50a3f0
I20260926 09:33:01.751744 8429 ts_tablet_manager.cc:1918] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:33:01.751801 8429 log.cc:1196] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/0059ef52a79e4654a540949d2b50a3f0
I20260926 09:33:01.751956 8431 ts_tablet_manager.cc:1939] T 0059ef52a79e4654a540949d2b50a3f0 P 731039aab732446994aa11f3e7a9b226: Deleting consensus metadata
I20260926 09:33:01.752029 8429 ts_tablet_manager.cc:1939] T 0059ef52a79e4654a540949d2b50a3f0 P 15614bf72be54ebc8615d15ddad1ac47: Deleting consensus metadata
I20260926 09:33:01.752345 8408 tablet_bootstrap.cc:492] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: Bootstrap starting.
I20260926 09:33:01.752424 7881 catalog_manager.cc:4841] TS 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247): tablet 0059ef52a79e4654a540949d2b50a3f0 (table test-table [id=0856e5c800224f9cad8ee743a2b9ccce]) successfully deleted
I20260926 09:33:01.752446 7880 catalog_manager.cc:4841] TS 731039aab732446994aa11f3e7a9b226 (127.0.98.3:40255): tablet 0059ef52a79e4654a540949d2b50a3f0 (table test-table [id=0856e5c800224f9cad8ee743a2b9ccce]) successfully deleted
I20260926 09:33:01.752514 8032 tablet_service.cc:1467] Processing CreateTablet for tablet a1dd2c67e4c5489c99e8ebe8302bdd60 (DEFAULT_TABLE table=test-table [id=4e12ceba2e754c65b69a6f8fc0fb3c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:01.752619 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1dd2c67e4c5489c99e8ebe8302bdd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.752622 8430 tablet_replica.cc:330] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: stopping tablet replica
I20260926 09:33:01.752688 8430 raft_consensus.cc:2229] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:01.752759 8430 raft_consensus.cc:2258] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:01.752760 8408 tablet_bootstrap.cc:654] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.753183 8430 ts_tablet_manager.cc:1905] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:01.753361 8408 tablet_bootstrap.cc:492] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: No bootstrap required, opened a new log
I20260926 09:33:01.753399 8408 ts_tablet_manager.cc:1397] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:01.753530 8408 raft_consensus.cc:348] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.753588 8408 raft_consensus.cc:374] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.753607 8408 raft_consensus.cc:729] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.753652 8408 consensus_queue.cc:260] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.753748 8408 ts_tablet_manager.cc:1428] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.753851 8409 tablet_bootstrap.cc:492] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: Bootstrap starting.
I20260926 09:33:01.754263 8409 tablet_bootstrap.cc:654] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.754783 8430 ts_tablet_manager.cc:1918] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:33:01.754858 8430 log.cc:1196] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/0059ef52a79e4654a540949d2b50a3f0
I20260926 09:33:01.754918 8409 tablet_bootstrap.cc:492] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: No bootstrap required, opened a new log
I20260926 09:33:01.754969 8409 ts_tablet_manager.cc:1397] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:01.755031 8171 tablet_service.cc:1467] Processing CreateTablet for tablet a1dd2c67e4c5489c99e8ebe8302bdd60 (DEFAULT_TABLE table=test-table [id=4e12ceba2e754c65b69a6f8fc0fb3c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:01.755115 8430 ts_tablet_manager.cc:1939] T 0059ef52a79e4654a540949d2b50a3f0 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting consensus metadata
I20260926 09:33:01.755134 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1dd2c67e4c5489c99e8ebe8302bdd60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.755112 8409 raft_consensus.cc:348] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.755187 8409 raft_consensus.cc:374] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.755214 8409 raft_consensus.cc:729] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15614bf72be54ebc8615d15ddad1ac47, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.755261 8409 consensus_queue.cc:260] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.755402 8409 ts_tablet_manager.cc:1428] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:33:01.755841 7879 catalog_manager.cc:4841] TS 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841): tablet 0059ef52a79e4654a540949d2b50a3f0 (table test-table [id=0856e5c800224f9cad8ee743a2b9ccce]) successfully deleted
I20260926 09:33:01.756350 8406 tablet_bootstrap.cc:492] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: Bootstrap starting.
I20260926 09:33:01.756660 8406 tablet_bootstrap.cc:654] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.757140 8406 tablet_bootstrap.cc:492] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: No bootstrap required, opened a new log
I20260926 09:33:01.757174 8406 ts_tablet_manager.cc:1397] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:01.757285 8406 raft_consensus.cc:348] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.757341 8406 raft_consensus.cc:374] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.757354 8406 raft_consensus.cc:729] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055c9c7afaf44ddf9433d7ade3abafea, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.757416 8406 consensus_queue.cc:260] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.757509 8406 ts_tablet_manager.cc:1428] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260926 09:33:01.827088 8099 tablet.cc:2369] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:33:01.865490 8414 raft_consensus.cc:482] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:33:01.865609 8414 raft_consensus.cc:504] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.865963 8414 leader_election.cc:288] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841), 731039aab732446994aa11f3e7a9b226 (127.0.98.3:40255)
I20260926 09:33:01.869035 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.866301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48815 (local address 127.0.98.3:40255)
0926 09:33:01.866349 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.866351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.866480 (+ 129us) server_negotiation.cc:407] Connection header received
0926 09:33:01.866573 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.866574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.866584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.866635 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:01.866815 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.867252 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.868313 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.868685 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.868739 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.868748 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.868917 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.868961 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:33:01.869099 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.866351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51449 (local address 127.0.98.2:32841)
0926 09:33:01.866397 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.866399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.866411 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:33:01.866487 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.866488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.866497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.866555 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:01.866739 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.867239 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.868351 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.868697 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.868785 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.868795 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.868988 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.869033 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:33:01.869077 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.866166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40255 (local address 127.0.98.1:48815)
0926 09:33:01.866418 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.866429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.866493 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.866652 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.866652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.866655 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:01.866736 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.866738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.867374 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.867374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.868223 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.868225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.868765 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.868766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.868839 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.868848 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.868890 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.868919 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":54,"thread_start_us":67,"threads_started":1}
I20260926 09:33:01.869105 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.866166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32841 (local address 127.0.98.1:51449)
0926 09:33:01.866286 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.866321 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.866395 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.866573 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.866574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.866577 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:01.866655 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.866658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.867351 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.867352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.868230 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.868232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.868802 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.868803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.868891 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.868896 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.868960 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.868997 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":37,"thread_start_us":66,"threads_started":1}
I20260926 09:33:01.869681 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1dd2c67e4c5489c99e8ebe8302bdd60" candidate_uuid: "15614bf72be54ebc8615d15ddad1ac47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea" is_pre_election: true
I20260926 09:33:01.869781 8192 raft_consensus.cc:2454] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15614bf72be54ebc8615d15ddad1ac47 in term 0.
I20260926 09:33:01.869735 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1dd2c67e4c5489c99e8ebe8302bdd60" candidate_uuid: "15614bf72be54ebc8615d15ddad1ac47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731039aab732446994aa11f3e7a9b226" is_pre_election: true
I20260926 09:33:01.869892 8332 raft_consensus.cc:2454] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15614bf72be54ebc8615d15ddad1ac47 in term 0.
I20260926 09:33:01.869978 7983 leader_election.cc:302] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055c9c7afaf44ddf9433d7ade3abafea, 15614bf72be54ebc8615d15ddad1ac47; no voters:
I20260926 09:33:01.870115 8414 raft_consensus.cc:2781] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:33:01.870157 8414 raft_consensus.cc:482] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:33:01.870177 8414 raft_consensus.cc:3037] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:01.870844 8414 raft_consensus.cc:504] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.871002 8414 leader_election.cc:288] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [CANDIDATE]: Term 1 election: Requested vote from peers 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841), 731039aab732446994aa11f3e7a9b226 (127.0.98.3:40255)
I20260926 09:33:01.871131 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1dd2c67e4c5489c99e8ebe8302bdd60" candidate_uuid: "15614bf72be54ebc8615d15ddad1ac47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731039aab732446994aa11f3e7a9b226"
I20260926 09:33:01.871143 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1dd2c67e4c5489c99e8ebe8302bdd60" candidate_uuid: "15614bf72be54ebc8615d15ddad1ac47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea"
I20260926 09:33:01.871212 8332 raft_consensus.cc:3037] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:01.871222 8192 raft_consensus.cc:3037] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:01.871829 8192 raft_consensus.cc:2454] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15614bf72be54ebc8615d15ddad1ac47 in term 1.
I20260926 09:33:01.871945 8332 raft_consensus.cc:2454] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15614bf72be54ebc8615d15ddad1ac47 in term 1.
I20260926 09:33:01.872092 7983 leader_election.cc:302] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055c9c7afaf44ddf9433d7ade3abafea, 15614bf72be54ebc8615d15ddad1ac47; no voters:
I20260926 09:33:01.872200 8414 raft_consensus.cc:2781] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:01.872340 8414 raft_consensus.cc:686] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 LEADER]: Becoming Leader. State: Replica: 15614bf72be54ebc8615d15ddad1ac47, State: Running, Role: LEADER
I20260926 09:33:01.872411 8414 consensus_queue.cc:237] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.873023 7894 catalog_manager.cc:5495] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "15614bf72be54ebc8615d15ddad1ac47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } health_report { overall_health: UNKNOWN } } }
W20260926 09:33:01.893934 8379 tablet.cc:2369] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 09:33:01 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:33:01.934966 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.928164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.0.1:34670)
0926 09:33:01.928346 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:01.928361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:01.928452 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:01.928558 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:01.928559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:01.928857 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:01.928904 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.928906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.929606 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.929607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.930143 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:01.930144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.930482 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.930483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:01.930521 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.931003 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:01.931007 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:01.932370 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:01.934090 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.934096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.934098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.934279 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.934454 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:01.934455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:01.934455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:01.934509 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:01.934687 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:01.934692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:01.934729 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:33:01.934779 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:33:01.934864 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260926 09:33:01.935011 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:01.928186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.0.98.62:41845)
0926 09:33:01.928254 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:01.928256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:01.928440 (+ 184us) server_negotiation.cc:407] Connection header received
0926 09:33:01.928495 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:01.928496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:01.928504 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:01.928553 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:01.928968 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.929538 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.930221 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:01.930411 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:01.930465 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:01.932453 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:01.932454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:01.932454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:01.932461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:01.934017 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.934343 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.934344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.934344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.934397 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:01.934548 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:01.934548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:01.934549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:01.934643 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:01.934692 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:01.934797 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 09:33:01.934909 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":41}
I20260926 09:33:01.937242 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34670:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:33:01.937466 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:33:01.939020 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 47dbb20cee2b454d81fe67943844de3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6c4701370fe4258aa3ec6c84963a134]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:33:01.939002 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 47dbb20cee2b454d81fe67943844de3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6c4701370fe4258aa3ec6c84963a134]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:33:01.939033 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 47dbb20cee2b454d81fe67943844de3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6c4701370fe4258aa3ec6c84963a134]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:33:01.939169 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dbb20cee2b454d81fe67943844de3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.939169 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dbb20cee2b454d81fe67943844de3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.939199 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dbb20cee2b454d81fe67943844de3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:01.940425 8406 tablet_bootstrap.cc:492] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: Bootstrap starting.
I20260926 09:33:01.940690 8408 tablet_bootstrap.cc:492] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: Bootstrap starting.
I20260926 09:33:01.940893 8406 tablet_bootstrap.cc:654] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.940896 8409 tablet_bootstrap.cc:492] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: Bootstrap starting.
I20260926 09:33:01.941358 8409 tablet_bootstrap.cc:654] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.941622 8406 tablet_bootstrap.cc:492] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: No bootstrap required, opened a new log
I20260926 09:33:01.941671 8406 ts_tablet_manager.cc:1397] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:01.941836 8406 raft_consensus.cc:348] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.941928 8406 raft_consensus.cc:374] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.941954 8406 raft_consensus.cc:729] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055c9c7afaf44ddf9433d7ade3abafea, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.941959 8408 tablet_bootstrap.cc:654] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:01.942018 8406 consensus_queue.cc:260] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.942128 8406 ts_tablet_manager.cc:1428] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.942204 8410 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:01.942335 8410 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:33:01.942447 8409 tablet_bootstrap.cc:492] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: No bootstrap required, opened a new log
I20260926 09:33:01.942535 8409 ts_tablet_manager.cc:1397] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:33:01.942679 8409 raft_consensus.cc:348] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.942760 8409 raft_consensus.cc:374] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.942782 8409 raft_consensus.cc:729] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15614bf72be54ebc8615d15ddad1ac47, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.942824 8409 consensus_queue.cc:260] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.942917 8409 ts_tablet_manager.cc:1428] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:33:01.943038 8414 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:01.943107 8414 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:33:01.943639 8408 tablet_bootstrap.cc:492] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: No bootstrap required, opened a new log
I20260926 09:33:01.943699 8408 ts_tablet_manager.cc:1397] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:33:01.943807 8408 raft_consensus.cc:348] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.943873 8408 raft_consensus.cc:374] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:01.943887 8408 raft_consensus.cc:729] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Initialized, Role: FOLLOWER
I20260926 09:33:01.943943 8408 consensus_queue.cc:260] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:01.944036 8408 ts_tablet_manager.cc:1428] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:01.944052 8416 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:01.944119 8416 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:33:02.035452 8416 raft_consensus.cc:482] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:33:02.035600 8416 raft_consensus.cc:504] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.035787 8416 leader_election.cc:288] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247), 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:02.035989 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dbb20cee2b454d81fe67943844de3f" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea" is_pre_election: true
I20260926 09:33:02.036034 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dbb20cee2b454d81fe67943844de3f" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15614bf72be54ebc8615d15ddad1ac47" is_pre_election: true
I20260926 09:33:02.036123 8192 raft_consensus.cc:2454] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 0.
I20260926 09:33:02.036115 8052 raft_consensus.cc:2454] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 0.
I20260926 09:33:02.036273 8265 leader_election.cc:302] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15614bf72be54ebc8615d15ddad1ac47, 731039aab732446994aa11f3e7a9b226; no voters:
I20260926 09:33:02.036412 8416 raft_consensus.cc:2781] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:33:02.036474 8416 raft_consensus.cc:482] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:33:02.036496 8416 raft_consensus.cc:3037] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.037197 8416 raft_consensus.cc:504] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.037335 8416 leader_election.cc:288] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 election: Requested vote from peers 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247), 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:02.037503 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dbb20cee2b454d81fe67943844de3f" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15614bf72be54ebc8615d15ddad1ac47"
I20260926 09:33:02.037505 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dbb20cee2b454d81fe67943844de3f" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea"
I20260926 09:33:02.037603 8192 raft_consensus.cc:3037] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.037606 8052 raft_consensus.cc:3037] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.038123 8192 raft_consensus.cc:2454] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 1.
I20260926 09:33:02.038298 8263 leader_election.cc:302] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055c9c7afaf44ddf9433d7ade3abafea, 731039aab732446994aa11f3e7a9b226; no voters:
I20260926 09:33:02.038335 8052 raft_consensus.cc:2454] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 1.
I20260926 09:33:02.038403 8416 raft_consensus.cc:2781] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:02.038452 8416 raft_consensus.cc:686] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [term 1 LEADER]: Becoming Leader. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Running, Role: LEADER
I20260926 09:33:02.038504 8416 consensus_queue.cc:237] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.038815 8424 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: TxnStatusTablet state changed. Reason: New leader 731039aab732446994aa11f3e7a9b226. Latest consensus state: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:02.038910 8424 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:33:02.039116 8449 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:33:02.039247 8449 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:33:02.039170 7894 catalog_manager.cc:5495] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731039aab732446994aa11f3e7a9b226 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } health_report { overall_health: HEALTHY } } }
I20260926 09:33:02.061811 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.059046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.0.98.3:40255)
0926 09:33:02.059094 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:02.059100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:02.059140 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:33:02.059219 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:02.059220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:02.059231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:02.059281 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:33:02.059469 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.059940 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.061104 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.061280 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.061419 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.061569 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.061599 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.061642 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:02.061730 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 09:33:02.061750 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:33:02.061873 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.058988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40255 (local address 127.0.0.1:46312)
0926 09:33:02.059090 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:02.059103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:02.059156 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:02.059339 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:02.059340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:02.059342 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:33:02.059395 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:02.059398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.060018 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.060019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:02.061025 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:02.061029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.061384 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.061385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:02.061455 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.061483 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.061665 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.061674 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 09:33:02.061725 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:33:02.061776 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 09:33:02.063006 8192 raft_consensus.cc:1261] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Refusing update from remote peer 731039aab732446994aa11f3e7a9b226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:02.063012 8052 raft_consensus.cc:1261] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Refusing update from remote peer 731039aab732446994aa11f3e7a9b226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:02.063254 8424 consensus_queue.cc:1035] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:02.063391 8416 consensus_queue.cc:1035] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:02.064023 8414 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: TxnStatusTablet state changed. Reason: New leader 731039aab732446994aa11f3e7a9b226. Latest consensus state: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:02.064136 8414 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:33:02.064741 8413 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:02.064824 8413 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:33:02.065029 8410 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: TxnStatusTablet state changed. Reason: New leader 731039aab732446994aa11f3e7a9b226. Latest consensus state: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:02.065143 8410 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:33:02.065670 8410 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:02.065768 8410 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 055c9c7afaf44ddf9433d7ade3abafea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:33:02.065892 8421 mvcc.cc:205] Tried to move back new op lower bound from 7333540585728217088 to 7333540585630113792. Current Snapshot: MvccSnapshot[applied={T|T < 7333540585728217088}]
I20260926 09:33:02.065995 8414 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:02.066068 8414 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 15614bf72be54ebc8615d15ddad1ac47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:33:02.066262 8416 tablet_replica.cc:439] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } } }
I20260926 09:33:02.066334 8416 tablet_replica.cc:442] T 47dbb20cee2b454d81fe67943844de3f P 731039aab732446994aa11f3e7a9b226: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:33:02.069085 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.066344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.0.98.1:41247)
0926 09:33:02.066386 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:02.066387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:02.066395 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:33:02.066503 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:02.066504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:02.066526 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:02.066593 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:33:02.066710 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.067477 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.068197 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.068509 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.068579 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.068741 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.068875 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.068948 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:02.068988 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:33:02.069011 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:33:02.074471 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.071811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55253 (local address 127.0.98.3:40255)
0926 09:33:02.071882 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:02.071883 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:02.071972 (+ 89us) server_negotiation.cc:407] Connection header received
0926 09:33:02.072033 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:02.072033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:02.072044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:02.072090 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:33:02.072225 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.072657 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.073827 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.074005 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.074052 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.074256 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.074280 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.074338 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:02.074395 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 09:33:02.074413 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:33:02.074530 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.071724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40255 (local address 127.0.98.1:55253)
0926 09:33:02.071892 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:02.071910 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:02.071971 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:02.072097 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:02.072098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:02.072099 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:33:02.072169 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:02.072170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.072727 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.072728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:02.073744 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:02.073747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.074096 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.074097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:02.074158 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.074174 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.074339 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.074341 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:33:02.074380 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 09:33:02.074419 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260926 09:33:02.076757 8192 raft_consensus.cc:1261] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Refusing update from remote peer 15614bf72be54ebc8615d15ddad1ac47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:02.076778 8332 raft_consensus.cc:1261] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Refusing update from remote peer 15614bf72be54ebc8615d15ddad1ac47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:02.077009 8414 consensus_queue.cc:1035] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:02.077319 8437 consensus_queue.cc:1035] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:02.086110 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.082791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51655 (local address 127.0.98.1:41247)
0926 09:33:02.082826 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:02.082828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:02.082864 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:33:02.082939 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:02.082940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:02.082951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:02.083012 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:33:02.083140 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.083704 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.085263 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.085539 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.085607 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.085833 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.085861 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.085952 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:02.086010 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 09:33:02.086033 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:33:02.086134 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.082753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41247 (local address 127.0.98.3:51655)
0926 09:33:02.082822 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:02.082833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:02.082875 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:02.083031 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:02.083033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:02.083034 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:33:02.083082 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:02.083084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.084599 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.084600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:02.085165 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:02.085167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.085679 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.085680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:02.085744 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.085760 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.085954 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.085955 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:33:02.086005 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:33:02.086051 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:33:02.093107 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:02.093219 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:02.093940 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: Sending DeleteTablet for 3 replicas of tablet a1dd2c67e4c5489c99e8ebe8302bdd60
I20260926 09:33:02.094201 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet a1dd2c67e4c5489c99e8ebe8302bdd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37318
I20260926 09:33:02.094218 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet a1dd2c67e4c5489c99e8ebe8302bdd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46296
I20260926 09:33:02.094306 8431 tablet_replica.cc:330] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: stopping tablet replica
I20260926 09:33:02.094322 8430 tablet_replica.cc:330] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: stopping tablet replica
I20260926 09:33:02.094349 8431 raft_consensus.cc:2229] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:02.094373 8430 raft_consensus.cc:2229] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:02.094379 8431 raft_consensus.cc:2258] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.094408 8430 raft_consensus.cc:2258] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.095053 8431 ts_tablet_manager.cc:1905] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:02.095620 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet a1dd2c67e4c5489c99e8ebe8302bdd60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46670
I20260926 09:33:02.095724 8429 tablet_replica.cc:330] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: stopping tablet replica
I20260926 09:33:02.095788 8429 raft_consensus.cc:2229] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:33:02.095878 8429 raft_consensus.cc:2258] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.095943 8430 ts_tablet_manager.cc:1905] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:02.096204 8429 ts_tablet_manager.cc:1905] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:02.096915 8431 ts_tablet_manager.cc:1918] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:33:02.096971 8431 log.cc:1196] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/a1dd2c67e4c5489c99e8ebe8302bdd60
I20260926 09:33:02.097209 8431 ts_tablet_manager.cc:1939] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 731039aab732446994aa11f3e7a9b226: Deleting consensus metadata
I20260926 09:33:02.097529 8430 ts_tablet_manager.cc:1918] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:33:02.097585 8430 log.cc:1196] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/a1dd2c67e4c5489c99e8ebe8302bdd60
I20260926 09:33:02.097803 8429 ts_tablet_manager.cc:1918] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:33:02.097833 8430 ts_tablet_manager.cc:1939] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting consensus metadata
I20260926 09:33:02.097851 8429 log.cc:1196] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/a1dd2c67e4c5489c99e8ebe8302bdd60
I20260926 09:33:02.098058 8429 ts_tablet_manager.cc:1939] T a1dd2c67e4c5489c99e8ebe8302bdd60 P 15614bf72be54ebc8615d15ddad1ac47: Deleting consensus metadata
I20260926 09:33:02.098158 7879 catalog_manager.cc:4841] TS 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841): tablet a1dd2c67e4c5489c99e8ebe8302bdd60 (table test-table [id=4e12ceba2e754c65b69a6f8fc0fb3c43]) successfully deleted
I20260926 09:33:02.098292 7880 catalog_manager.cc:4841] TS 731039aab732446994aa11f3e7a9b226 (127.0.98.3:40255): tablet a1dd2c67e4c5489c99e8ebe8302bdd60 (table test-table [id=4e12ceba2e754c65b69a6f8fc0fb3c43]) successfully deleted
I20260926 09:33:02.098433 7881 catalog_manager.cc:4841] TS 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247): tablet a1dd2c67e4c5489c99e8ebe8302bdd60 (table test-table [id=4e12ceba2e754c65b69a6f8fc0fb3c43]) successfully deleted
Sep 26 09:33:02 dist-test-slave-21h8 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:33:02 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:02 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:33:02.126606 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:02.118763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.0.98.62:41845)
0926 09:33:02.118814 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:02.118816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:02.119526 (+ 710us) server_negotiation.cc:407] Connection header received
0926 09:33:02.119590 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:02.119591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:02.119603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:02.119671 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:02.120446 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.121231 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.122318 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.122636 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.122709 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.126535 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34672: BlockingRecv error: recv got EOF from 127.0.0.1:34672 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 09:33:02 dist-test-slave-21h8 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:33:02.140777 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.127751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.0.98.62:41845)
0926 09:33:02.127798 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:02.127800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:02.131544 (+ 3744us) server_negotiation.cc:407] Connection header received
0926 09:33:02.131621 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:02.131622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:02.131634 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:02.131698 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:02.132336 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.133137 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.134190 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.134507 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.134579 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.137520 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:02.137522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:02.137522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:02.137531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:02.139452 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:02.139867 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:02.139868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:02.139869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:02.139939 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:02.140176 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:02.140177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:02.140177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:02.140318 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:02.140428 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:02.140575 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 09:33:02.140668 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:33:02.142537 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:33:02.142724 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:33:02.144182 8171 tablet_service.cc:1467] Processing CreateTablet for tablet b9336342ffe84a65b5a5cc9e3190f038 (DEFAULT_TABLE table=test-table [id=5abe784f85d74cb6a5e943d9f76ea21c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:02.144205 8032 tablet_service.cc:1467] Processing CreateTablet for tablet b9336342ffe84a65b5a5cc9e3190f038 (DEFAULT_TABLE table=test-table [id=5abe784f85d74cb6a5e943d9f76ea21c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:02.144330 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9336342ffe84a65b5a5cc9e3190f038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:02.144351 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9336342ffe84a65b5a5cc9e3190f038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:02.145833 8409 tablet_bootstrap.cc:492] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: Bootstrap starting.
I20260926 09:33:02.145833 8406 tablet_bootstrap.cc:492] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: Bootstrap starting.
I20260926 09:33:02.146236 8406 tablet_bootstrap.cc:654] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:02.146286 8409 tablet_bootstrap.cc:654] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:02.146648 8309 tablet_service.cc:1467] Processing CreateTablet for tablet b9336342ffe84a65b5a5cc9e3190f038 (DEFAULT_TABLE table=test-table [id=5abe784f85d74cb6a5e943d9f76ea21c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:02.146756 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9336342ffe84a65b5a5cc9e3190f038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:02.147001 8406 tablet_bootstrap.cc:492] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: No bootstrap required, opened a new log
I20260926 09:33:02.147044 8406 ts_tablet_manager.cc:1397] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:02.147189 8406 raft_consensus.cc:348] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.147258 8406 raft_consensus.cc:374] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:02.147277 8406 raft_consensus.cc:729] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055c9c7afaf44ddf9433d7ade3abafea, State: Initialized, Role: FOLLOWER
I20260926 09:33:02.147363 8406 consensus_queue.cc:260] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.147486 8406 ts_tablet_manager.cc:1428] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:33:02.148121 8408 tablet_bootstrap.cc:492] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: Bootstrap starting.
I20260926 09:33:02.148509 8408 tablet_bootstrap.cc:654] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:02.150640 8408 tablet_bootstrap.cc:492] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: No bootstrap required, opened a new log
I20260926 09:33:02.150695 8408 ts_tablet_manager.cc:1397] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:33:02.150815 8408 raft_consensus.cc:348] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.150874 8408 raft_consensus.cc:374] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:02.150893 8408 raft_consensus.cc:729] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Initialized, Role: FOLLOWER
I20260926 09:33:02.150933 8408 consensus_queue.cc:260] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.151016 8408 ts_tablet_manager.cc:1428] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:02.151679 8409 tablet_bootstrap.cc:492] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: No bootstrap required, opened a new log
I20260926 09:33:02.151729 8409 ts_tablet_manager.cc:1397] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260926 09:33:02.151856 8409 raft_consensus.cc:348] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.151916 8409 raft_consensus.cc:374] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:02.151937 8409 raft_consensus.cc:729] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15614bf72be54ebc8615d15ddad1ac47, State: Initialized, Role: FOLLOWER
I20260926 09:33:02.151976 8409 consensus_queue.cc:260] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.152064 8409 ts_tablet_manager.cc:1428] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:02.255496 8416 raft_consensus.cc:482] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:33:02.255602 8416 raft_consensus.cc:504] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.255824 8416 leader_election.cc:288] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247), 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:02.256076 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9336342ffe84a65b5a5cc9e3190f038" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15614bf72be54ebc8615d15ddad1ac47" is_pre_election: true
I20260926 09:33:02.256076 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9336342ffe84a65b5a5cc9e3190f038" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea" is_pre_election: true
I20260926 09:33:02.256178 8052 raft_consensus.cc:2454] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 0.
I20260926 09:33:02.256178 8192 raft_consensus.cc:2454] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 0.
I20260926 09:33:02.256367 8263 leader_election.cc:302] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055c9c7afaf44ddf9433d7ade3abafea, 731039aab732446994aa11f3e7a9b226; no voters:
I20260926 09:33:02.256474 8416 raft_consensus.cc:2781] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:33:02.256521 8416 raft_consensus.cc:482] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:33:02.256552 8416 raft_consensus.cc:3037] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.257236 8416 raft_consensus.cc:504] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.257381 8416 leader_election.cc:288] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 election: Requested vote from peers 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247), 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:02.257598 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9336342ffe84a65b5a5cc9e3190f038" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15614bf72be54ebc8615d15ddad1ac47"
I20260926 09:33:02.257706 8052 raft_consensus.cc:3037] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.257679 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9336342ffe84a65b5a5cc9e3190f038" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea"
I20260926 09:33:02.257743 8192 raft_consensus.cc:3037] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.258476 8192 raft_consensus.cc:2454] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 1.
I20260926 09:33:02.258481 8052 raft_consensus.cc:2454] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 1.
I20260926 09:33:02.258679 8265 leader_election.cc:302] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15614bf72be54ebc8615d15ddad1ac47, 731039aab732446994aa11f3e7a9b226; no voters:
I20260926 09:33:02.258788 8416 raft_consensus.cc:2781] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:02.258842 8416 raft_consensus.cc:686] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 1 LEADER]: Becoming Leader. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Running, Role: LEADER
I20260926 09:33:02.258889 8416 consensus_queue.cc:237] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.259505 7894 catalog_manager.cc:5495] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731039aab732446994aa11f3e7a9b226 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } health_report { overall_health: HEALTHY } } }
I20260926 09:33:02.268959 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.266883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.0.98.3:40255)
0926 09:33:02.266957 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:02.266959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:02.266991 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:33:02.267010 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:02.267011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:02.267019 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:02.267059 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:33:02.267168 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.267688 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.268410 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.268581 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.268630 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.268759 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.268784 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.268821 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:02.268883 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 09:33:02.268898 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:33:02.269730 8192 raft_consensus.cc:1261] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Refusing update from remote peer 731039aab732446994aa11f3e7a9b226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:02.269728 8052 raft_consensus.cc:1261] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Refusing update from remote peer 731039aab732446994aa11f3e7a9b226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:02.269995 8416 consensus_queue.cc:1035] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:02.270102 8413 consensus_queue.cc:1035] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:02.272426 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:02.272523 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:02.273120 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: Sending DeleteTablet for 3 replicas of tablet b9336342ffe84a65b5a5cc9e3190f038
I20260926 09:33:02.273384 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet b9336342ffe84a65b5a5cc9e3190f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46670
I20260926 09:33:02.273429 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet b9336342ffe84a65b5a5cc9e3190f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37318
I20260926 09:33:02.273502 8429 tablet_replica.cc:330] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: stopping tablet replica
I20260926 09:33:02.273519 8430 tablet_replica.cc:330] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: stopping tablet replica
I20260926 09:33:02.273576 8429 raft_consensus.cc:2229] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:02.273579 8430 raft_consensus.cc:2229] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:02.273633 8429 raft_consensus.cc:2258] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.273651 8430 raft_consensus.cc:2258] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.273635 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet b9336342ffe84a65b5a5cc9e3190f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46296
I20260926 09:33:02.273794 8431 tablet_replica.cc:330] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: stopping tablet replica
I20260926 09:33:02.273850 8431 raft_consensus.cc:2229] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:33:02.273926 8431 raft_consensus.cc:2258] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.274088 8429 ts_tablet_manager.cc:1905] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:02.274039 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:33:02.274205 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:33:02.274520 8431 ts_tablet_manager.cc:1905] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:02.275419 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2cf47aad6f48b3840fc1573c57e67e (DEFAULT_TABLE table=test-table [id=d567c37f0e39420f991f63d1d7d8e8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:02.275447 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2cf47aad6f48b3840fc1573c57e67e (DEFAULT_TABLE table=test-table [id=d567c37f0e39420f991f63d1d7d8e8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:02.275533 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2cf47aad6f48b3840fc1573c57e67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:02.275554 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2cf47aad6f48b3840fc1573c57e67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:02.275843 8429 ts_tablet_manager.cc:1918] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:33:02.275892 8429 log.cc:1196] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/b9336342ffe84a65b5a5cc9e3190f038
I20260926 09:33:02.276014 8431 ts_tablet_manager.cc:1918] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:33:02.276064 8431 log.cc:1196] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/b9336342ffe84a65b5a5cc9e3190f038
I20260926 09:33:02.276144 8429 ts_tablet_manager.cc:1939] T b9336342ffe84a65b5a5cc9e3190f038 P 15614bf72be54ebc8615d15ddad1ac47: Deleting consensus metadata
I20260926 09:33:02.276289 8431 ts_tablet_manager.cc:1939] T b9336342ffe84a65b5a5cc9e3190f038 P 731039aab732446994aa11f3e7a9b226: Deleting consensus metadata
I20260926 09:33:02.277002 8408 tablet_bootstrap.cc:492] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: Bootstrap starting.
I20260926 09:33:02.277081 7881 catalog_manager.cc:4841] TS 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247): tablet b9336342ffe84a65b5a5cc9e3190f038 (table test-table [id=5abe784f85d74cb6a5e943d9f76ea21c]) successfully deleted
I20260926 09:33:02.277158 8406 tablet_bootstrap.cc:492] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: Bootstrap starting.
I20260926 09:33:02.277339 8408 tablet_bootstrap.cc:654] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:02.277724 8406 tablet_bootstrap.cc:654] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:02.277853 8408 tablet_bootstrap.cc:492] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: No bootstrap required, opened a new log
I20260926 09:33:02.277886 8408 ts_tablet_manager.cc:1397] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:02.277961 8408 raft_consensus.cc:348] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.278028 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2cf47aad6f48b3840fc1573c57e67e (DEFAULT_TABLE table=test-table [id=d567c37f0e39420f991f63d1d7d8e8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:02.278054 8408 raft_consensus.cc:374] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:02.278081 8408 raft_consensus.cc:729] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Initialized, Role: FOLLOWER
I20260926 09:33:02.278105 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2cf47aad6f48b3840fc1573c57e67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:02.278115 8408 consensus_queue.cc:260] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.278225 8408 ts_tablet_manager.cc:1428] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:02.278393 8406 tablet_bootstrap.cc:492] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: No bootstrap required, opened a new log
I20260926 09:33:02.278430 8406 ts_tablet_manager.cc:1397] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:33:02.278573 8406 raft_consensus.cc:348] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.278641 8406 raft_consensus.cc:374] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:02.278661 8406 raft_consensus.cc:729] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055c9c7afaf44ddf9433d7ade3abafea, State: Initialized, Role: FOLLOWER
I20260926 09:33:02.278723 8406 consensus_queue.cc:260] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.278825 8406 ts_tablet_manager.cc:1428] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:02.279366 8409 tablet_bootstrap.cc:492] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47: Bootstrap starting.
I20260926 09:33:02.279647 8409 tablet_bootstrap.cc:654] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:02.280054 8430 ts_tablet_manager.cc:1905] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:02.280184 8409 tablet_bootstrap.cc:492] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47: No bootstrap required, opened a new log
I20260926 09:33:02.280230 8409 ts_tablet_manager.cc:1397] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:33:02.280337 8409 raft_consensus.cc:348] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.280391 8409 raft_consensus.cc:374] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:02.280411 8409 raft_consensus.cc:729] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15614bf72be54ebc8615d15ddad1ac47, State: Initialized, Role: FOLLOWER
I20260926 09:33:02.280467 8409 consensus_queue.cc:260] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.280544 8409 ts_tablet_manager.cc:1428] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:02.281571 8430 ts_tablet_manager.cc:1918] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:33:02.281638 8430 log.cc:1196] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/b9336342ffe84a65b5a5cc9e3190f038
I20260926 09:33:02.281853 8430 ts_tablet_manager.cc:1939] T b9336342ffe84a65b5a5cc9e3190f038 P 055c9c7afaf44ddf9433d7ade3abafea: Deleting consensus metadata
I20260926 09:33:02.281836 7880 catalog_manager.cc:4841] TS 731039aab732446994aa11f3e7a9b226 (127.0.98.3:40255): tablet b9336342ffe84a65b5a5cc9e3190f038 (table test-table [id=5abe784f85d74cb6a5e943d9f76ea21c]) successfully deleted
I20260926 09:33:02.282119 7879 catalog_manager.cc:4841] TS 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841): tablet b9336342ffe84a65b5a5cc9e3190f038 (table test-table [id=5abe784f85d74cb6a5e943d9f76ea21c]) successfully deleted
I20260926 09:33:02.288357 8424 raft_consensus.cc:482] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:33:02.288479 8424 raft_consensus.cc:504] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.288662 8424 leader_election.cc:288] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247), 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:02.288827 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2cf47aad6f48b3840fc1573c57e67e" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15614bf72be54ebc8615d15ddad1ac47" is_pre_election: true
I20260926 09:33:02.288861 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2cf47aad6f48b3840fc1573c57e67e" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea" is_pre_election: true
I20260926 09:33:02.288925 8052 raft_consensus.cc:2454] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 0.
I20260926 09:33:02.288936 8192 raft_consensus.cc:2454] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 0.
I20260926 09:33:02.289103 8263 leader_election.cc:302] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055c9c7afaf44ddf9433d7ade3abafea, 731039aab732446994aa11f3e7a9b226; no voters:
I20260926 09:33:02.289204 8424 raft_consensus.cc:2781] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:33:02.289248 8424 raft_consensus.cc:482] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:33:02.289276 8424 raft_consensus.cc:3037] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.289929 8424 raft_consensus.cc:504] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.290072 8424 leader_election.cc:288] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 election: Requested vote from peers 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247), 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841)
I20260926 09:33:02.290211 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2cf47aad6f48b3840fc1573c57e67e" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15614bf72be54ebc8615d15ddad1ac47"
I20260926 09:33:02.290277 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2cf47aad6f48b3840fc1573c57e67e" candidate_uuid: "731039aab732446994aa11f3e7a9b226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055c9c7afaf44ddf9433d7ade3abafea"
I20260926 09:33:02.290321 8052 raft_consensus.cc:3037] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.290354 8192 raft_consensus.cc:3037] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:02.291042 8192 raft_consensus.cc:2454] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 1.
I20260926 09:33:02.291042 8052 raft_consensus.cc:2454] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731039aab732446994aa11f3e7a9b226 in term 1.
I20260926 09:33:02.291224 8263 leader_election.cc:302] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055c9c7afaf44ddf9433d7ade3abafea, 731039aab732446994aa11f3e7a9b226; no voters:
I20260926 09:33:02.291353 8424 raft_consensus.cc:2781] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:02.291430 8424 raft_consensus.cc:686] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 1 LEADER]: Becoming Leader. State: Replica: 731039aab732446994aa11f3e7a9b226, State: Running, Role: LEADER
I20260926 09:33:02.291495 8424 consensus_queue.cc:237] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } }
I20260926 09:33:02.292045 7895 catalog_manager.cc:5495] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731039aab732446994aa11f3e7a9b226 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "731039aab732446994aa11f3e7a9b226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731039aab732446994aa11f3e7a9b226" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40255 } health_report { overall_health: HEALTHY } } }
I20260926 09:33:02.301265 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.299249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55177 (local address 127.0.98.3:40255)
0926 09:33:02.299286 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:02.299288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:02.299295 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:33:02.299393 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:02.299394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:02.299402 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:02.299451 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:33:02.299557 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.300017 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.300696 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.300875 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.300925 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.301034 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.301060 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.301104 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:02.301151 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 09:33:02.301177 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":19}
I20260926 09:33:02.301268 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.299185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40255 (local address 127.0.98.3:55177)
0926 09:33:02.299242 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:02.299255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:02.299296 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:02.299457 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:02.299458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:02.299459 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:33:02.299503 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:02.299505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.300082 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.300082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:02.300622 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:02.300624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:02.300938 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:02.300938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:02.300974 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:02.300978 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:33:02.301115 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:33:02.301116 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:33:02.301150 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 09:33:02.301193 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":3}
I20260926 09:33:02.303184 8052 raft_consensus.cc:1261] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Refusing update from remote peer 731039aab732446994aa11f3e7a9b226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:02.303190 8192 raft_consensus.cc:1261] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Refusing update from remote peer 731039aab732446994aa11f3e7a9b226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:02.303455 8413 consensus_queue.cc:1035] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055c9c7afaf44ddf9433d7ade3abafea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:02.303573 8424 consensus_queue.cc:1035] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15614bf72be54ebc8615d15ddad1ac47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:02.313756 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:02.313876 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:02.314476 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1910f3ab77f43e69590eb6a6f605489: Sending DeleteTablet for 3 replicas of tablet 6b2cf47aad6f48b3840fc1573c57e67e
I20260926 09:33:02.314718 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b2cf47aad6f48b3840fc1573c57e67e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46296
I20260926 09:33:02.314762 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b2cf47aad6f48b3840fc1573c57e67e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46670
I20260926 09:33:02.314765 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b2cf47aad6f48b3840fc1573c57e67e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37318
I20260926 09:33:02.314827 8431 tablet_replica.cc:330] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: stopping tablet replica
I20260926 09:33:02.314852 8429 tablet_replica.cc:330] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47: stopping tablet replica
I20260926 09:33:02.314875 8430 tablet_replica.cc:330] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: stopping tablet replica
I20260926 09:33:02.314904 8429 raft_consensus.cc:2229] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:02.314893 8431 raft_consensus.cc:2229] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:33:02.314915 8430 raft_consensus.cc:2229] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:02.314939 8429 raft_consensus.cc:2258] T 6b2cf47aad6f48b3840fc1573c57e67e P 15614bf72be54ebc8615d15ddad1ac47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.314946 8430 raft_consensus.cc:2258] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.314965 8431 raft_consensus.cc:2258] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:02.315152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 7967
I20260926 09:33:02.315276 8431 ts_tablet_manager.cc:1905] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:02.317063 8431 ts_tablet_manager.cc:1918] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:33:02.317126 8431 log.cc:1196] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/6b2cf47aad6f48b3840fc1573c57e67e
I20260926 09:33:02.317337 8431 ts_tablet_manager.cc:1939] T 6b2cf47aad6f48b3840fc1573c57e67e P 731039aab732446994aa11f3e7a9b226: Deleting consensus metadata
I20260926 09:33:02.317723 7880 catalog_manager.cc:4841] TS 731039aab732446994aa11f3e7a9b226 (127.0.98.3:40255): tablet 6b2cf47aad6f48b3840fc1573c57e67e (table test-table [id=d567c37f0e39420f991f63d1d7d8e8fd]) successfully deleted
I20260926 09:33:02.317797 8430 ts_tablet_manager.cc:1905] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:02.318929 8430 ts_tablet_manager.cc:1918] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:33:02.319008 8430 log.cc:1196] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/6b2cf47aad6f48b3840fc1573c57e67e
I20260926 09:33:02.319214 8430 ts_tablet_manager.cc:1939] T 6b2cf47aad6f48b3840fc1573c57e67e P 055c9c7afaf44ddf9433d7ade3abafea: Deleting consensus metadata
I20260926 09:33:02.319582 7879 catalog_manager.cc:4841] TS 055c9c7afaf44ddf9433d7ade3abafea (127.0.98.2:32841): tablet 6b2cf47aad6f48b3840fc1573c57e67e (table test-table [id=d567c37f0e39420f991f63d1d7d8e8fd]) successfully deleted
W20260926 09:33:02.322289 7881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41247 (error 108)
I20260926 09:33:02.322590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 8107
I20260926 09:33:02.323387 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:02.322631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41247 (local address 127.0.0.1:46682)
0926 09:33:02.323269 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:02.323340 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":43,"threads_started":1}
W20260926 09:33:02.324107 7881 catalog_manager.cc:4584] TS 15614bf72be54ebc8615d15ddad1ac47 (127.0.98.1:41247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b2cf47aad6f48b3840fc1573c57e67e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41247: connect: Connection refused (error 111)
I20260926 09:33:02.328051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 8247
I20260926 09:33:02.334204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 7863
2026-09-26T09:33:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2529 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:33:02 dist-test-slave-21h8 krb5kdc[8493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:33:02 dist-test-slave-21h8 krb5kdc[8493](info): set up 2 sockets
Sep 26 09:33:02 dist-test-slave-21h8 krb5kdc[8493](info): commencing operation
krb5kdc: starting...
W20260926 09:33:03.383879 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:33:03 dist-test-slave-21h8 krb5kdc[8493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:33:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:33:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:33:03.502163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:03.582588 8506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:03.582798 8506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:03.582820 8506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:03.584060 8506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:33:03.584105 8506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:03.584120 8506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:03.584131 8506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:33:03.584143 8506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:33:03.585620 8506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:03.585881 8506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:03.586115 8506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:03.590878 8512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:03.590883 8511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:03.590934 8506 server_base.cc:1034] running on GCE node
W20260926 09:33:03.590878 8514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:03.591305 8506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:03.591672 8506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:03.592814 8506 hybrid_clock.cc:648] HybridClock initialized: now 1790415183592800 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T09:33:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1264 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:33:03 dist-test-slave-21h8 krb5kdc[8519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:33:03 dist-test-slave-21h8 krb5kdc[8519](info): set up 2 sockets
Sep 26 09:33:03 dist-test-slave-21h8 krb5kdc[8519](info): commencing operation
krb5kdc: starting...
W20260926 09:33:04.652017 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:33:04 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:33:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:33:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:33:04.773922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44461
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:04.856719 8532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:04.856925 8532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:04.856947 8532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:04.858120 8532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:33:04.858165 8532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:04.858179 8532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:04.858192 8532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:33:04.858245 8532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:33:04.859841 8532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:04.860105 8532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:04.860332 8532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:04.865206 8540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:04.865231 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:04.865206 8538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:04.865410 8532 server_base.cc:1034] running on GCE node
I20260926 09:33:04.865712 8532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:04.866029 8532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:04.867199 8532 hybrid_clock.cc:648] HybridClock initialized: now 1790415184867184 us; error 44 us; skew 500 ppm
Sep 26 09:33:04 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:04.877599 8532 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:33:04.878077 8532 webserver.cc:460] Webserver started at http://127.0.98.62:41609/ using document root <none> and password file <none>
I20260926 09:33:04.878291 8532 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:04.878335 8532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:04.878427 8532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:04.879621 8532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "32e1d2bb99ab47a59cd82aae471e9dec"
format_stamp: "Formatted at 2026-09-26 09:33:04 on dist-test-slave-21h8"
server_key: "ede14b7d976bec377e9d1bc8129a0a22"
server_key_iv: "1d18b1705e90a5b1a80201bd97030fa5"
server_key_version: "encryptionkey@0"
I20260926 09:33:04.879920 8532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "32e1d2bb99ab47a59cd82aae471e9dec"
format_stamp: "Formatted at 2026-09-26 09:33:04 on dist-test-slave-21h8"
server_key: "ede14b7d976bec377e9d1bc8129a0a22"
server_key_iv: "1d18b1705e90a5b1a80201bd97030fa5"
server_key_version: "encryptionkey@0"
I20260926 09:33:04.881222 8532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:33:04.881989 8547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:04.882184 8532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:04.882300 8532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "32e1d2bb99ab47a59cd82aae471e9dec"
format_stamp: "Formatted at 2026-09-26 09:33:04 on dist-test-slave-21h8"
server_key: "ede14b7d976bec377e9d1bc8129a0a22"
server_key_iv: "1d18b1705e90a5b1a80201bd97030fa5"
server_key_version: "encryptionkey@0"
I20260926 09:33:04.882377 8532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:04.896306 8532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:04.898959 8532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:04.899091 8532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:04.903059 8532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44461
I20260926 09:33:04.903189 8599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44461 every 8 connection(s)
I20260926 09:33:04.903702 8532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:33:04.904508 8600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:04.906837 8600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Bootstrap starting.
I20260926 09:33:04.907579 8600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:04.907907 8600 log.cc:824] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:04.908627 8600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: No bootstrap required, opened a new log
I20260926 09:33:04.909116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 8532
I20260926 09:33:04.909420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:33:04.909639 392 external_mini_cluster.cc:1428] Setting key c7cb6157bd41c61d54b731e238b02008
I20260926 09:33:04.911656 8600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } }
I20260926 09:33:04.911844 8600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:04.911888 8600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e1d2bb99ab47a59cd82aae471e9dec, State: Initialized, Role: FOLLOWER
I20260926 09:33:04.912052 8600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } }
I20260926 09:33:04.912158 8600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:33:04.912204 8600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:33:04.912276 8600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:04.912912 8600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } }
I20260926 09:33:04.913040 8600 leader_election.cc:302] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32e1d2bb99ab47a59cd82aae471e9dec; no voters:
I20260926 09:33:04.913259 8600 leader_election.cc:288] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:33:04.913314 8605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:04.913463 8605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 1 LEADER]: Becoming Leader. State: Replica: 32e1d2bb99ab47a59cd82aae471e9dec, State: Running, Role: LEADER
I20260926 09:33:04.913558 8600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:33:04.913620 8605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } }
I20260926 09:33:04.914122 8606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } } }
I20260926 09:33:04.914223 8606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:04.914494 8607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32e1d2bb99ab47a59cd82aae471e9dec. Latest consensus state: current_term: 1 leader_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } } }
I20260926 09:33:04.914630 8607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:04.914816 8614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:33:04.915467 8614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:33:04.917523 8614 catalog_manager.cc:1347] Generated new cluster ID: 76c53acab4914fa4820baca4c913b032
I20260926 09:33:04.917593 8614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:33:04 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:33:04.922761 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:04.910238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.0.98.62:44461)
0926 09:33:04.910444 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:04.910454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:04.910489 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:33:04.911582 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:04.911590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:04.911644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:04.911807 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:04.912627 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:04.913812 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:04.915036 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:04.915754 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:04.915877 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:04.918774 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:04.918779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:04.918779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:04.918792 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:04.920971 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:04.921385 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:04.921389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:04.921389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:04.921534 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:04.921765 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:04.921766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:04.921767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:04.922121 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:04.922275 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:04.922390 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 09:33:04.922540 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260926 09:33:04.933049 8614 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:33:04.933610 8614 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:33:04.939832 8614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Generated new TSK 0
I20260926 09:33:04.940084 8614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:33:04.977908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44461
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:05.085285 8626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:05.085520 8626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:05.085559 8626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:05.086851 8626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:05.086922 8626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:05.086968 8626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:33:05.088568 8626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:05.088897 8626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:05.089135 8626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:05.094175 8634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.094192 8632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.094182 8631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:05.094587 8626 server_base.cc:1034] running on GCE node
I20260926 09:33:05.094815 8626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:05.095037 8626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:05.096218 8626 hybrid_clock.cc:648] HybridClock initialized: now 1790415185096204 us; error 31 us; skew 500 ppm
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:05.098829 8626 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:33:05.099272 8626 webserver.cc:460] Webserver started at http://127.0.98.1:44855/ using document root <none> and password file <none>
I20260926 09:33:05.099551 8626 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:05.099625 8626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:05.099777 8626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:05.100867 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "9524f45b792b4514a924e7e69722c68c"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "5abb9b0780d8c20914574dd56228a8d0"
server_key_iv: "4f6c310856b0789dc18941d29fc07129"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.101225 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9524f45b792b4514a924e7e69722c68c"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "5abb9b0780d8c20914574dd56228a8d0"
server_key_iv: "4f6c310856b0789dc18941d29fc07129"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.102631 8626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:33:05.103471 8641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.103648 8626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:05.103747 8626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "9524f45b792b4514a924e7e69722c68c"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "5abb9b0780d8c20914574dd56228a8d0"
server_key_iv: "4f6c310856b0789dc18941d29fc07129"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.103832 8626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:05.117801 8626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:05.119841 8626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:05.120007 8626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:05.120249 8626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:05.120568 8626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:05.120622 8626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.120666 8626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:05.120690 8626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.126618 8626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37049
I20260926 09:33:05.126669 8754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37049 every 8 connection(s)
I20260926 09:33:05.126981 8626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:33:05.133788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 8626
I20260926 09:33:05.133900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:33:05.134119 392 external_mini_cluster.cc:1428] Setting key 7091b12daaf2e8233e7d67ff480282fa
I20260926 09:33:05.137104 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.127715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45565 (local address 127.0.98.62:44461)
0926 09:33:05.127804 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.127807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.128022 (+ 215us) server_negotiation.cc:407] Connection header received
0926 09:33:05.128937 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.128938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.128949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.129005 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.129613 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.130113 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.131100 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.131374 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.131453 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.133706 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:05.133708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:05.133709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:05.133717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:05.135419 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:05.135957 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:05.135957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:05.135958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:05.136010 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:05.136345 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:05.136347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:05.136347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:05.136522 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:05.136603 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:05.136871 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 09:33:05.136969 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:33:05.137127 8757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.127689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:45565)
0926 09:33:05.127889 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.127926 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.128839 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.129079 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.129083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.129339 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.129528 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.129532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.130194 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.130195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.131011 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.131012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.131471 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.131472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.131578 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.132165 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.132173 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.133608 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:05.135498 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:05.135505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:05.135508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:05.135864 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:05.136073 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:05.136074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:05.136075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:05.136259 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:05.136598 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:05.136603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:05.136727 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 09:33:05.136827 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 09:33:05.136956 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":124,"threads_started":1}
I20260926 09:33:05.137984 8755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44461
I20260926 09:33:05.138106 8755 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:05.138309 8755 heartbeater.cc:511] Master 127.0.98.62:44461 requested a full tablet report, sending...
I20260926 09:33:05.138957 8564 ts_manager.cc:194] Registered new tserver with Master: 9524f45b792b4514a924e7e69722c68c (127.0.98.1:37049)
I20260926 09:33:05.140025 8564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45565
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:33:05.160295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44461
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:05.242583 8760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:05.242811 8760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:05.242837 8760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:05.244014 8760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:05.244051 8760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:05.244086 8760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:33:05.245639 8760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:05.245857 8760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:05.246058 8760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:05.251139 8768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.251137 8765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.251113 8766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:05.251309 8760 server_base.cc:1034] running on GCE node
I20260926 09:33:05.251641 8760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:05.251906 8760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:05.253086 8760 hybrid_clock.cc:648] HybridClock initialized: now 1790415185253062 us; error 41 us; skew 500 ppm
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:05.255852 8760 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:33:05.256371 8760 webserver.cc:460] Webserver started at http://127.0.98.2:38113/ using document root <none> and password file <none>
I20260926 09:33:05.256618 8760 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:05.256688 8760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:05.256848 8760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:05.257921 8760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4646fbc49694efa86dea01e9c9744ae"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "95de6a6eb40951ee5021b8a83326975a"
server_key_iv: "88d3deb75806f84d3c9c47c1bd1b55a8"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.258306 8760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4646fbc49694efa86dea01e9c9744ae"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "95de6a6eb40951ee5021b8a83326975a"
server_key_iv: "88d3deb75806f84d3c9c47c1bd1b55a8"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.259747 8760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:33:05.260527 8775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.260816 8760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:05.260923 8760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "c4646fbc49694efa86dea01e9c9744ae"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "95de6a6eb40951ee5021b8a83326975a"
server_key_iv: "88d3deb75806f84d3c9c47c1bd1b55a8"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.261013 8760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:05.275354 8760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:05.277974 8760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:05.278147 8760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:05.278394 8760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:05.278750 8760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:05.278806 8760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.278856 8760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:05.278883 8760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.285269 8760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37141
I20260926 09:33:05.285400 8888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37141 every 8 connection(s)
I20260926 09:33:05.285777 8760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:33:05.295598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 8760
I20260926 09:33:05.295730 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.286655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35949 (local address 127.0.98.62:44461)
0926 09:33:05.286725 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.286727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.286979 (+ 252us) server_negotiation.cc:407] Connection header received
0926 09:33:05.288013 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.288014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.288027 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.288082 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.288767 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.289275 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.290341 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.290549 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.290626 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.292957 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:05.292958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:05.292958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:05.292967 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:05.294195 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:05.294624 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:05.294626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:05.294626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:05.294691 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:05.294965 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:05.294966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:05.294966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:05.295097 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:05.295168 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:05.295545 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 09:33:05.295630 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:33:05.295733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:33:05.295945 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.286662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:35949)
0926 09:33:05.286847 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.286884 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.287891 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.288171 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.288174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.288464 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.288665 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.288669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.289379 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.289380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.290215 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.290217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.290634 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.290636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.290765 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.291267 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.291276 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.292798 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:05.294283 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:05.294291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:05.294294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:05.294544 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:05.294766 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:05.294767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:05.294767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:05.294891 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:05.295168 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:05.295175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:05.295387 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 09:33:05.295526 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 09:33:05.295697 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260926 09:33:05.295991 392 external_mini_cluster.cc:1428] Setting key bff440449e237bc47a0b9282190cbd70
I20260926 09:33:05.296586 8889 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44461
I20260926 09:33:05.296720 8889 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:05.296936 8889 heartbeater.cc:511] Master 127.0.98.62:44461 requested a full tablet report, sending...
I20260926 09:33:05.297315 8564 ts_manager.cc:194] Registered new tserver with Master: c4646fbc49694efa86dea01e9c9744ae (127.0.98.2:37141)
I20260926 09:33:05.297708 8564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35949
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:33:05.323181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44461
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:05.412093 8894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:05.412348 8894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:05.412388 8894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:05.413651 8894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:05.413714 8894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:05.413760 8894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:33:05.415673 8894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:05.415963 8894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:05.416200 8894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:05.421216 8900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:05.421406 8894 server_base.cc:1034] running on GCE node
W20260926 09:33:05.421247 8899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.421267 8902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:05.421711 8894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:05.422006 8894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:05.423177 8894 hybrid_clock.cc:648] HybridClock initialized: now 1790415185423161 us; error 42 us; skew 500 ppm
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:05.425627 8894 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:33:05.426033 8894 webserver.cc:460] Webserver started at http://127.0.98.3:40423/ using document root <none> and password file <none>
I20260926 09:33:05.426218 8894 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:05.426258 8894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:05.426348 8894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:05.427416 8894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "56ed48cc658a4305a90058db3009e4cc"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "9437124dd20b47db855d4d78a94843b0"
server_key_iv: "cb2d4f503c3444ab83b89f51308e9d22"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.427768 8894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56ed48cc658a4305a90058db3009e4cc"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "9437124dd20b47db855d4d78a94843b0"
server_key_iv: "cb2d4f503c3444ab83b89f51308e9d22"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.429047 8894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:33:05.429711 8909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.429883 8894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:33:05.429976 8894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "56ed48cc658a4305a90058db3009e4cc"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "9437124dd20b47db855d4d78a94843b0"
server_key_iv: "cb2d4f503c3444ab83b89f51308e9d22"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.430032 8894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:05.437523 8894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:05.439769 8894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:05.439953 8894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:05.440187 8894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:05.440501 8894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:05.440536 8894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.440557 8894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:05.440605 8894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.446085 8894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33847
I20260926 09:33:05.446164 9022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33847 every 8 connection(s)
I20260926 09:33:05.446468 8894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:33:05.448057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 8894
I20260926 09:33:05.448132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:33:05.448295 392 external_mini_cluster.cc:1428] Setting key be1d3867f8216df1af7767528362699a
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:33:05.455910 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.447236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59187 (local address 127.0.98.62:44461)
0926 09:33:05.447296 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.447298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.447555 (+ 257us) server_negotiation.cc:407] Connection header received
0926 09:33:05.448584 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.448586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.448598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.448654 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.449436 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.449957 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.450954 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.451172 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.451248 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.453295 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:05.453308 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:05.453308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:05.453316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:05.454473 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:05.454879 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:05.454881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:05.454881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:05.454943 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:05.455193 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:05.455194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:05.455194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:05.455354 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:05.455437 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:05.455735 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 09:33:05.455814 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:33:05.456110 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.447215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:59187)
0926 09:33:05.447416 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.447459 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.448496 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.448731 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.448735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.449084 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.449349 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.449354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.450053 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.450054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.450856 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.450858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.451253 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.451254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.451426 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.451863 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.451871 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.453200 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:05.454566 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:05.454572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:05.454573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:05.454791 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:05.455011 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:05.455012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:05.455012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:05.455123 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:05.455429 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:05.455434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:05.455602 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 09:33:05.455713 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 09:33:05.455870 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":120,"threads_started":1}
I20260926 09:33:05.456620 9023 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44461
I20260926 09:33:05.456724 9023 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:05.456862 9023 heartbeater.cc:511] Master 127.0.98.62:44461 requested a full tablet report, sending...
I20260926 09:33:05.457218 8564 ts_manager.cc:194] Registered new tserver with Master: 56ed48cc658a4305a90058db3009e4cc (127.0.98.3:33847)
I20260926 09:33:05.457546 8564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59187
I20260926 09:33:05.460139 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 09:33:05.534193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 8626
I20260926 09:33:05.538776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 8760
I20260926 09:33:05.543035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 8894
I20260926 09:33:05.547240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 8532
I20260926 09:33:05.552348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44461
--webserver_interface=127.0.98.62
--webserver_port=41609
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:05.634760 9032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:05.635015 9032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:05.635054 9032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:05.636271 9032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:33:05.636343 9032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:05.636368 9032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:05.636410 9032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:33:05.636436 9032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:33:05.637971 9032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:05.638252 9032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:05.638490 9032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:05.643347 9037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.643407 9040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:05.643380 9032 server_base.cc:1034] running on GCE node
W20260926 09:33:05.643338 9038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:05.643852 9032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:05.644179 9032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:05.645352 9032 hybrid_clock.cc:648] HybridClock initialized: now 1790415185645349 us; error 39 us; skew 500 ppm
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:05.648000 9032 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:33:05.648439 9032 webserver.cc:460] Webserver started at http://127.0.98.62:41609/ using document root <none> and password file <none>
I20260926 09:33:05.648679 9032 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:05.648761 9032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:05.650094 9032 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:05.650729 9047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.650899 9032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:33:05.651010 9032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "32e1d2bb99ab47a59cd82aae471e9dec"
format_stamp: "Formatted at 2026-09-26 09:33:04 on dist-test-slave-21h8"
server_key: "ede14b7d976bec377e9d1bc8129a0a22"
server_key_iv: "1d18b1705e90a5b1a80201bd97030fa5"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.651418 9032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:05.674860 9032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:05.677413 9032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:05.677584 9032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:05.681514 9032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44461
I20260926 09:33:05.681571 9099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44461 every 8 connection(s)
I20260926 09:33:05.682125 9032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:33:05.683470 9100 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:33:05.684139 9100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Bootstrap starting.
I20260926 09:33:05.687228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 9032
I20260926 09:33:05.687556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37049
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44461
--webserver_port=44855
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:33:05.692816 9100 log.cc:824] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:05.694526 9100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:33:05.694837 9100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Bootstrap complete.
I20260926 09:33:05.697157 9100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } }
I20260926 09:33:05.697391 9100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e1d2bb99ab47a59cd82aae471e9dec, State: Initialized, Role: FOLLOWER
I20260926 09:33:05.697562 9100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } }
I20260926 09:33:05.697635 9100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:33:05.697685 9100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:33:05.697731 9100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:33:05.698746 9100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } }
I20260926 09:33:05.698860 9100 leader_election.cc:302] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32e1d2bb99ab47a59cd82aae471e9dec; no voters:
I20260926 09:33:05.699062 9100 leader_election.cc:288] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:33:05.699098 9104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:33:05.699208 9104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [term 2 LEADER]: Becoming Leader. State: Replica: 32e1d2bb99ab47a59cd82aae471e9dec, State: Running, Role: LEADER
I20260926 09:33:05.699383 9100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:33:05.699391 9104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } }
I20260926 09:33:05.699818 9106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32e1d2bb99ab47a59cd82aae471e9dec. Latest consensus state: current_term: 2 leader_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } } }
I20260926 09:33:05.699910 9106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:05.700193 9105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1d2bb99ab47a59cd82aae471e9dec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44461 } } }
I20260926 09:33:05.700266 9105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:05.700418 9110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:33:05.700999 9110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:33:05.701231 9110 catalog_manager.cc:1259] Loaded cluster ID: 76c53acab4914fa4820baca4c913b032
I20260926 09:33:05.701445 9110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:33:05.702267 9110 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:33:05.702463 9110 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 32e1d2bb99ab47a59cd82aae471e9dec: Loaded TSK: 0
I20260926 09:33:05.702679 9110 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:33:05.786712 9102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:05.786921 9102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:05.786944 9102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:05.788208 9102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:05.788252 9102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:05.788280 9102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:33:05.789691 9102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44855
--tserver_master_addrs=127.0.98.62:44461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:05.789950 9102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:05.790194 9102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:05.795254 9129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.795285 9126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.795260 9127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:05.795773 9102 server_base.cc:1034] running on GCE node
I20260926 09:33:05.796003 9102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:05.796268 9102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:05.797427 9102 hybrid_clock.cc:648] HybridClock initialized: now 1790415185797406 us; error 39 us; skew 500 ppm
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:05.800523 9102 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 09:33:05.801080 9102 webserver.cc:460] Webserver started at http://127.0.98.1:44855/ using document root <none> and password file <none>
I20260926 09:33:05.801344 9102 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:05.801424 9102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:05.803236 9102 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:33:05.803983 9136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.804206 9102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:05.804306 9102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "9524f45b792b4514a924e7e69722c68c"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "5abb9b0780d8c20914574dd56228a8d0"
server_key_iv: "4f6c310856b0789dc18941d29fc07129"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.804780 9102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:05.818099 9102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:05.820827 9102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:05.820984 9102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:05.821255 9102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:05.821619 9102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:05.821666 9102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.821698 9102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:05.821724 9102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.829485 9102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37049
I20260926 09:33:05.829515 9249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37049 every 8 connection(s)
I20260926 09:33:05.829962 9102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:33:05.832937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 9102
I20260926 09:33:05.833262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37141
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44461
--webserver_port=38113
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:05.840257 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.830678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:34157)
0926 09:33:05.830894 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.830942 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.831616 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.832068 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.832072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.832837 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.833093 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.833099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.834765 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.834766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.835741 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.835743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.836364 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.836365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.836490 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.837091 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.837102 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.840051 (+ 2949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":111,"threads_started":1}
W20260926 09:33:05.840700 9250 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44461 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44461: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260926 09:33:05.841020 9253 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:05.830835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34157 (local address 127.0.98.62:44461)
0926 09:33:05.830995 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.831005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.831039 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:33:05.831819 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.831825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.831868 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.831986 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.833227 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.834660 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.835839 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.836268 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.836408 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.840235 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34157: BlockingRecv error: recv got EOF from 127.0.98.1:34157 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:05.847005 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.840967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:54879)
0926 09:33:05.841326 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.841339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.841402 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.841588 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.841589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.841754 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.841817 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.841819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.842806 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.842807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.843767 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.843770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.844269 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.844270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.844350 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.844834 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.844840 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.846928 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342}
W20260926 09:33:05.847273 9255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:05.841009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54879 (local address 127.0.98.62:44461)
0926 09:33:05.841153 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.841155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.841383 (+ 228us) server_negotiation.cc:407] Connection header received
0926 09:33:05.841468 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.841469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.841480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.841567 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.841911 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.842706 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.843861 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.844177 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.844255 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.847037 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54879: BlockingRecv error: recv got EOF from 127.0.98.1:54879 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:05.852725 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.847388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:35837)
0926 09:33:05.847427 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.847437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.847471 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.847700 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.847701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.847838 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.847870 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.847871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.848749 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.848750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.849567 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.849569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.850044 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.850045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.850098 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.850590 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.850598 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.852655 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 09:33:05.852963 9253 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:05.847477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35837 (local address 127.0.98.62:44461)
0926 09:33:05.847574 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.847576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.847584 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:33:05.847614 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.847615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.847624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.847685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.847955 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.848681 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.849657 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.849962 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.850037 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.852753 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35837: BlockingRecv error: recv got EOF from 127.0.98.1:35837 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 09:33:05.852982 9250 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:33:05.921140 9254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:05.921471 9254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:05.921514 9254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:05.922798 9254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:05.922868 9254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:05.922911 9254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:33:05.924425 9254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38113
--tserver_master_addrs=127.0.98.62:44461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:05.924710 9254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:05.924952 9254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:05.930084 9260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.930111 9263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:05.930084 9261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:05.930368 9254 server_base.cc:1034] running on GCE node
I20260926 09:33:05.930591 9254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:05.930872 9254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:05.932060 9254 hybrid_clock.cc:648] HybridClock initialized: now 1790415185932039 us; error 36 us; skew 500 ppm
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:05.934522 9254 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 09:33:05.934966 9254 webserver.cc:460] Webserver started at http://127.0.98.2:38113/ using document root <none> and password file <none>
I20260926 09:33:05.935222 9254 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:05.935307 9254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:05.936717 9254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:05.937296 9270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.937474 9254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:33:05.937577 9254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "c4646fbc49694efa86dea01e9c9744ae"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "95de6a6eb40951ee5021b8a83326975a"
server_key_iv: "88d3deb75806f84d3c9c47c1bd1b55a8"
server_key_version: "encryptionkey@0"
I20260926 09:33:05.937942 9254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:05.949082 9254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:05.950991 9254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:05.951155 9254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:05.951545 9254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:05.951875 9254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:05.951929 9254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.951964 9254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:05.952011 9254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:05.957996 9254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37141
I20260926 09:33:05.958069 9383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37141 every 8 connection(s)
I20260926 09:33:05.958419 9254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:33:05.959852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 9254
I20260926 09:33:05.960189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33847
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44461
--webserver_port=40423
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:33:05.966845 9253 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:05.959201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39177 (local address 127.0.98.62:44461)
0926 09:33:05.959254 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.959257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.959544 (+ 287us) server_negotiation.cc:407] Connection header received
0926 09:33:05.960076 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.960078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.960096 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.960178 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.961188 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.961922 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.963135 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.963393 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.963475 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.966793 (+ 3318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39177: BlockingRecv error: recv got EOF from 127.0.98.2:39177 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:33:05.966967 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.959212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:39177)
0926 09:33:05.959420 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.959458 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.959952 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.960238 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.960241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.960878 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.961086 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.961091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.962040 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.962042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.963036 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.963039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.963488 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.963489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.963632 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.964206 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.964215 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.966710 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
W20260926 09:33:05.967191 9384 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44461 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44461: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:05.972406 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.967476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:51267)
0926 09:33:05.967536 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.967548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.967596 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.967754 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.967755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.967874 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.967923 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.967924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.968918 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.968919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.969410 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.969411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.969909 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.969909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.969957 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.970455 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.970459 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.972341 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 09:33:05.972527 9253 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:05.967536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51267 (local address 127.0.98.62:44461)
0926 09:33:05.967612 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.967614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.967624 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:33:05.967651 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.967653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.967668 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.967752 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.967996 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.968827 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.969489 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.969825 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.969907 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.972461 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51267: BlockingRecv error: recv got EOF from 127.0.98.2:51267 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:05 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:05.977272 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:05.972677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:53941)
0926 09:33:05.972802 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:05.972813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:05.972852 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:05.972998 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:05.972999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:05.973102 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:05.973131 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.973132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.974046 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.974046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.974524 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:05.974526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.974960 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.974960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:05.974995 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.975451 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:05.975455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:05.977195 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
W20260926 09:33:05.977352 9253 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:05.972839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53941 (local address 127.0.98.62:44461)
0926 09:33:05.972874 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:05.972876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:05.972900 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:33:05.972924 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:05.972925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:05.972934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:05.972985 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:05.973195 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.973973 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.974599 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:05.974893 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:05.974957 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:05.977279 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53941: BlockingRecv error: recv got EOF from 127.0.98.2:53941 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 09:33:05.977408 9384 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:33:06.047683 9387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:06.047890 9387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:06.047914 9387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:06.049049 9387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:06.049088 9387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:06.049114 9387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:33:06.050508 9387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40423
--tserver_master_addrs=127.0.98.62:44461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:06.050699 9387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:06.050887 9387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:06.055903 9393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:06.055917 9394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:06.055886 9396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:06.056339 9387 server_base.cc:1034] running on GCE node
I20260926 09:33:06.056573 9387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:06.056885 9387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:06.058064 9387 hybrid_clock.cc:648] HybridClock initialized: now 1790415186058051 us; error 30 us; skew 500 ppm
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:06.060618 9387 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 09:33:06.061127 9387 webserver.cc:460] Webserver started at http://127.0.98.3:40423/ using document root <none> and password file <none>
I20260926 09:33:06.061378 9387 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:06.061480 9387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:06.062819 9387 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:06.063434 9403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:06.063606 9387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:06.063705 9387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "56ed48cc658a4305a90058db3009e4cc"
format_stamp: "Formatted at 2026-09-26 09:33:05 on dist-test-slave-21h8"
server_key: "9437124dd20b47db855d4d78a94843b0"
server_key_iv: "cb2d4f503c3444ab83b89f51308e9d22"
server_key_version: "encryptionkey@0"
I20260926 09:33:06.064069 9387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:06.073551 9387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:06.075544 9387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:06.075781 9387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:06.076040 9387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:06.076362 9387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:06.076416 9387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:06.076469 9387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:06.076493 9387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:06.082504 9387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33847
I20260926 09:33:06.082546 9516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33847 every 8 connection(s)
I20260926 09:33:06.082911 9387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:33:06.085299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 9387
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:06.091055 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:06.083725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:50193)
0926 09:33:06.083897 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:06.083933 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:06.084460 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:06.084727 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:06.084731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:06.085333 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:06.085605 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.085610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.086525 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.086526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.087475 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.087478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.087877 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.087878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.088020 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.088599 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:06.088612 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:06.090898 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
W20260926 09:33:06.091066 9253 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:06.083724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50193 (local address 127.0.98.62:44461)
0926 09:33:06.083767 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:06.083769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:06.084020 (+ 251us) server_negotiation.cc:407] Connection header received
0926 09:33:06.084583 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:06.084584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:06.084596 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:06.084647 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:06.085738 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.086424 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.087572 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.087777 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.087863 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.090998 (+ 3135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50193: BlockingRecv error: recv got EOF from 127.0.98.3:50193 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 09:33:06.091382 9517 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44461 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44461: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260926 09:33:06.093180 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:06.085854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.0.98.62:44461)
0926 09:33:06.085905 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:06.085907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:06.086049 (+ 142us) server_negotiation.cc:407] Connection header received
0926 09:33:06.086071 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:06.086072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:06.086084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:06.086134 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:06.086709 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.087190 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.088024 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.088213 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.088288 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.089483 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:06.089487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:06.089488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:06.089497 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:06.091835 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:06.092206 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:06.092210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:06.092210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:06.092301 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:06.092513 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:06.092514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:06.092515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:06.092721 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:06.092851 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:06.092966 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 09:33:06.093084 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:06.096202 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:06.091663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:35879)
0926 09:33:06.091714 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:06.091725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:06.091763 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:06.091876 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:06.091877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:06.092014 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:06.092058 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.092060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.092835 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.092836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.093591 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.093594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.094010 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.094011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.094082 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.094404 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:06.094407 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:06.096118 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 09:33:06.096246 9253 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:06.091716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35879 (local address 127.0.98.62:44461)
0926 09:33:06.091772 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:06.091776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:06.091788 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:33:06.091821 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:06.091822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:06.091830 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:06.091892 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:06.092128 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.092735 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.093666 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.093935 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.094000 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.096184 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35879: BlockingRecv error: recv got EOF from 127.0.98.3:35879 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:06.100692 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:06.096536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:33433)
0926 09:33:06.096592 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:06.096603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:06.096643 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:06.096763 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:06.096764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:06.096893 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:06.096918 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.096919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.097530 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.097531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.098027 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.098029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.098404 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.098405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.098455 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.098786 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:06.098791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:06.100609 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 09:33:06.100739 9253 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:06.096601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33433 (local address 127.0.98.62:44461)
0926 09:33:06.096640 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:06.096643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:06.096652 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:33:06.096696 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:06.096697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:06.096706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:06.096759 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:06.096993 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.097450 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.098119 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.098307 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.098379 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.100673 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33433: BlockingRecv error: recv got EOF from 127.0.98.3:33433 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:33:06.100888 9517 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:06.858107 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:06.853350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:50949)
0926 09:33:06.853531 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:06.853543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:06.853613 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:06.853765 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:06.853766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:06.853932 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:06.853982 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.853984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.854665 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.854666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.855235 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.855237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.855607 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.855608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.855648 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.856280 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:06.856285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:06.858036 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":99,"threads_started":1}
W20260926 09:33:06.858209 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:06.853442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50949 (local address 127.0.98.62:44461)
0926 09:33:06.853606 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:06.853608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:06.853617 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:33:06.853676 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:06.853677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:06.853687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:06.853749 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:06.854046 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.854536 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.855306 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.855543 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.855596 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.858131 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50949: BlockingRecv error: recv got EOF from 127.0.98.1:50949 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:06 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:06.982954 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:06.977862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:48611)
0926 09:33:06.978018 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:06.978036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:06.978095 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:06.978238 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:06.978239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:06.978402 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:06.978450 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.978451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.979118 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.979119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.979767 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:06.979769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.980215 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.980216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:06.980307 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.980949 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:06.980956 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:06.982877 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260926 09:33:06.983101 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:06.977959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48611 (local address 127.0.98.62:44461)
0926 09:33:06.978037 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:06.978039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:06.978082 (+ 43us) server_negotiation.cc:407] Connection header received
0926 09:33:06.978158 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:06.978159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:06.978169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:06.978212 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:06.978517 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.978999 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.979852 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:06.980114 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:06.980200 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:06.982997 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48611: BlockingRecv error: recv got EOF from 127.0.98.2:48611 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 09:33:07 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:07 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:07.106070 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:07.101299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:39383)
0926 09:33:07.101435 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:07.101447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:07.101512 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:07.101687 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:07.101688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:07.101858 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:07.101905 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:07.101907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.102621 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.102622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:07.103174 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:07.103176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.103570 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.103571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:07.103621 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:07.104186 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:07.104191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:07.105999 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
W20260926 09:33:07.106179 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:07.101362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39383 (local address 127.0.98.62:44461)
0926 09:33:07.101451 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:07.101453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:07.101562 (+ 109us) server_negotiation.cc:407] Connection header received
0926 09:33:07.101591 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:07.101592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:07.101606 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:07.101671 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:07.101999 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.102532 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.103274 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.103490 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.103554 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:07.106096 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39383: BlockingRecv error: recv got EOF from 127.0.98.3:39383 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 09:33:07 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:07 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:07.863665 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:07.858824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:59189)
0926 09:33:07.858973 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:07.858989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:07.859059 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:07.859230 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:07.859231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:07.859456 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:07.859504 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:07.859506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.860136 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.860136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:07.860778 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:07.860780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.861111 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.861112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:07.861149 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:07.861643 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:07.861648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:07.863555 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
W20260926 09:33:07.863711 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:07.858929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59189 (local address 127.0.98.62:44461)
0926 09:33:07.859083 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:07.859084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:07.859092 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:33:07.859135 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:07.859136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:07.859147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:07.859217 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:07.859576 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.860066 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.860849 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.861044 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.861099 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:07.863638 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59189: BlockingRecv error: recv got EOF from 127.0.98.1:59189 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
Sep 26 09:33:07 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:07 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:07.988359 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:07.983525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:40001)
0926 09:33:07.983661 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:07.983680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:07.983731 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:07.983881 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:07.983882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:07.984107 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:07.984158 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:07.984160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.984916 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.984917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:07.985478 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:07.985479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.985850 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.985852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:07.985920 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:07.986383 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:07.986387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:07.988252 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260926 09:33:07.988420 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:07.983635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40001 (local address 127.0.98.62:44461)
0926 09:33:07.983705 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:07.983707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:07.983728 (+ 21us) server_negotiation.cc:407] Connection header received
0926 09:33:07.983780 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:07.983781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:07.983791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:07.983857 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:07.984243 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.984812 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.985572 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:07.985763 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:07.985849 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:07.988340 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40001: BlockingRecv error: recv got EOF from 127.0.98.2:40001 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 09:33:08 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:08 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:08.111075 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:08.106560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:51121)
0926 09:33:08.106737 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:08.106748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:08.106799 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:08.106937 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:08.106938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:08.107098 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:08.107147 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:08.107148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.107829 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.107830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:08.108371 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:08.108373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.108804 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.108805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:08.108849 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:08.109294 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:08.109299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:08.111017 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
W20260926 09:33:08.111171 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:08.106705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51121 (local address 127.0.98.62:44461)
0926 09:33:08.106786 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:08.106787 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:08.106796 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:33:08.106855 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:08.106862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:08.106873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:08.106913 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:08.107211 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.107759 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.108503 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.108736 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.108791 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:08.111103 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51121: BlockingRecv error: recv got EOF from 127.0.98.3:51121 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 09:33:08 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:08 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:08.869127 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:08.864220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:49217)
0926 09:33:08.864362 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:08.864372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:08.864427 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:08.864570 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:08.864571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:08.864733 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:08.864781 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:08.864783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.865455 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.865456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:08.865996 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:08.866044 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.866402 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.866403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:08.866448 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:08.867088 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:08.867093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:08.869026 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
W20260926 09:33:08.869184 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:08.864284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49217 (local address 127.0.98.62:44461)
0926 09:33:08.864423 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:08.864431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:08.864466 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:33:08.864489 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:08.864489 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:08.864500 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:08.864558 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:08.864855 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.865363 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.866126 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.866333 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.866395 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:08.869111 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49217: BlockingRecv error: recv got EOF from 127.0.98.1:49217 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
Sep 26 09:33:08 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:08 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:08.993660 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:08.988904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:49471)
0926 09:33:08.989080 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:08.989095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:08.989190 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:08.989342 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:08.989343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:08.989538 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:08.989587 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:08.989589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.990282 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.990284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:08.990939 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:08.990941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.991297 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.991298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:08.991379 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:08.991860 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:08.991865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:08.993557 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
W20260926 09:33:08.993719 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:08.989003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49471 (local address 127.0.98.62:44461)
0926 09:33:08.989084 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:08.989086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:08.989170 (+ 84us) server_negotiation.cc:407] Connection header received
0926 09:33:08.989250 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:08.989251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:08.989263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:08.989334 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:08.989675 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.990187 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.991030 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:08.991217 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:08.991297 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:08.993653 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49471: BlockingRecv error: recv got EOF from 127.0.98.2:49471 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 09:33:09 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:09 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:09.116325 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:09.111590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:53795)
0926 09:33:09.111820 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:09.111831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:09.111925 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:09.112079 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:09.112080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:09.112272 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:09.112318 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:09.112321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.113005 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.113006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:09.113534 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:09.113536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.113902 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.113903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:09.113962 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:09.114413 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:09.114417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:09.116257 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
W20260926 09:33:09.116464 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:09.111674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53795 (local address 127.0.98.62:44461)
0926 09:33:09.111776 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:09.111779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:09.111907 (+ 128us) server_negotiation.cc:407] Connection header received
0926 09:33:09.111982 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:09.111983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:09.111998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:09.112073 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:09.112398 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.112911 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.113619 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.113813 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.113900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:09.116379 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53795: BlockingRecv error: recv got EOF from 127.0.98.3:53795 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 09:33:09 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:09 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:09.874683 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:09.869773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:57565)
0926 09:33:09.869986 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:09.869999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:09.870070 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:09.870261 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:09.870263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:09.870424 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:09.870473 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:09.870475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.871120 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.871121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:09.871737 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:09.871739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.872148 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.872149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:09.872209 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:09.872778 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:09.872783 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:09.874623 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":123,"threads_started":1}
W20260926 09:33:09.874805 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:09.869881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57565 (local address 127.0.98.62:44461)
0926 09:33:09.870042 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:09.870044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:09.870056 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:33:09.870131 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:09.870132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:09.870144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:09.870195 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:09.870546 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.871040 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.871847 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.872047 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.872113 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:09.874720 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57565: BlockingRecv error: recv got EOF from 127.0.98.1:57565 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
Sep 26 09:33:10 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:10 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:10.004792 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:09.994220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:54167)
0926 09:33:09.994366 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:09.994378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:09.994455 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:09.994608 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:09.994608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:09.994771 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:09.994820 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:09.994822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.995688 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.995689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:09.996331 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:09.996333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.996667 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.996668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:09.996711 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:10.002804 (+ 6093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:10.002811 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:10.004730 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
W20260926 09:33:10.004942 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:09.994345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54167 (local address 127.0.98.62:44461)
0926 09:33:09.994404 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:09.994406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:09.994457 (+ 51us) server_negotiation.cc:407] Connection header received
0926 09:33:09.994524 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:09.994525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:09.994536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:09.994587 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:09.994892 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.995586 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.996409 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:09.996595 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:09.996653 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:10.004859 (+ 8206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54167: BlockingRecv error: recv got EOF from 127.0.98.2:54167 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 09:33:10 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:10 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:10.121526 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:10.116819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:39379)
0926 09:33:10.116998 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:10.117013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:10.117089 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:10.117259 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:10.117260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:10.117451 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:10.117498 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:10.117500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:10.118168 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:10.118169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:10.118701 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:10.118703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:10.119078 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:10.119079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:10.119137 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:10.119637 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:10.119642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:10.121428 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
W20260926 09:33:10.121579 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:10.116943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39379 (local address 127.0.98.62:44461)
0926 09:33:10.117024 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:10.117027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:10.117067 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:33:10.117162 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:10.117163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:10.117177 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:10.117253 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:10.117579 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:10.118081 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:10.118788 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:10.119002 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:10.119079 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:10.121512 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39379: BlockingRecv error: recv got EOF from 127.0.98.3:39379 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 09:33:10 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:10 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:10.879709 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:10.875210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:37489)
0926 09:33:10.875391 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:10.875404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:10.875472 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:10.875606 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:10.875607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:10.875765 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:10.875814 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:10.875816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:10.876437 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:10.876437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:10.876985 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:10.876986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:10.877331 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:10.877332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:10.877369 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:10.877823 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:10.877828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:10.879616 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
W20260926 09:33:10.879840 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:10.875270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37489 (local address 127.0.98.62:44461)
0926 09:33:10.875427 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:10.875429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:10.875457 (+ 28us) server_negotiation.cc:407] Connection header received
0926 09:33:10.875526 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:10.875526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:10.875537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:10.875586 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:10.875879 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:10.876372 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:10.877071 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:10.877261 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:10.877317 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:10.879752 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37489: BlockingRecv error: recv got EOF from 127.0.98.1:37489 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
Sep 26 09:33:11 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:11 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:11.009907 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:11.005372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:34089)
0926 09:33:11.005564 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:11.005575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:11.005637 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:11.005784 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:11.005785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:11.005951 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:11.005998 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:11.006000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.006645 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.006646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:11.007184 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:11.007186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.007518 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.007519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:11.007558 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:11.007986 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:11.007990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:11.009815 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
W20260926 09:33:11.010098 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:11.005528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34089 (local address 127.0.98.62:44461)
0926 09:33:11.005589 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:11.005591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:11.005620 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:33:11.005696 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:11.005697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:11.005708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:11.005760 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:11.006056 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.006575 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.007254 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.007459 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.007510 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:11.009938 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34089: BlockingRecv error: recv got EOF from 127.0.98.2:34089 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 09:33:11 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:11 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:11.126579 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:11.122029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:49413)
0926 09:33:11.122173 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:11.122191 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:11.122266 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:11.122407 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:11.122408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:11.122572 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:11.122620 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:11.122622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.123229 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.123230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:11.123801 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:11.123804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.124116 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.124117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:11.124153 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:11.124667 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:11.124672 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:11.126511 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
W20260926 09:33:11.126698 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:11.122123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49413 (local address 127.0.98.62:44461)
0926 09:33:11.122197 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:11.122199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:11.122247 (+ 48us) server_negotiation.cc:407] Connection header received
0926 09:33:11.122325 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:11.122325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:11.122337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:11.122376 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:11.122686 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.123160 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.123874 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.124051 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.124105 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:11.126603 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49413: BlockingRecv error: recv got EOF from 127.0.98.3:49413 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 09:33:11 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:11 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:11.885090 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:11.880228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:53341)
0926 09:33:11.880406 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:11.880417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:11.880477 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:11.880618 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:11.880618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:11.880781 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:11.880830 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:11.880832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.881503 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.881504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:11.882010 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:11.882012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.882419 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.882420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:11.882468 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:11.883052 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:11.883057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:11.885028 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
W20260926 09:33:11.885263 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:11.880335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53341 (local address 127.0.98.62:44461)
0926 09:33:11.880469 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:11.880471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:11.880481 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:33:11.880527 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:11.880527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:11.880539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:11.880598 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:11.880899 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.881437 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.882102 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:11.882352 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:11.882423 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:11.885149 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53341: BlockingRecv error: recv got EOF from 127.0.98.1:53341 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":55,"threads_started":1}
Sep 26 09:33:12 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:12 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:12.015240 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:12.010381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:57385)
0926 09:33:12.010563 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:12.010575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:12.010667 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:12.010831 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:12.010832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:12.011025 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:12.011075 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:12.011077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.011834 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.011835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:12.012386 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:12.012388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.012750 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.012752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:12.012815 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:12.013318 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:12.013323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:12.015173 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
W20260926 09:33:12.015453 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:12.010482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57385 (local address 127.0.98.62:44461)
0926 09:33:12.010551 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:12.010553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:12.010647 (+ 94us) server_negotiation.cc:407] Connection header received
0926 09:33:12.010752 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:12.010753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:12.010763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:12.010825 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:12.011164 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.011740 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.012472 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.012667 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.012742 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:12.015360 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57385: BlockingRecv error: recv got EOF from 127.0.98.2:57385 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 09:33:12 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:12 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:12.132611 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:12.127187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:54987)
0926 09:33:12.127371 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:12.127394 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:12.127456 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:12.127601 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:12.127602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:12.127784 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:12.127836 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:12.127837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.128559 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.128560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:12.129130 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:12.129132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.129518 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.129520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:12.129560 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:12.130084 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:12.130088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:12.132549 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
W20260926 09:33:12.132751 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:12.127266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54987 (local address 127.0.98.62:44461)
0926 09:33:12.127353 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:12.127360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:12.127458 (+ 98us) server_negotiation.cc:407] Connection header received
0926 09:33:12.127503 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:12.127504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:12.127516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:12.127576 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:12.127919 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.128439 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.129232 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.129432 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.129503 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:12.132664 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54987: BlockingRecv error: recv got EOF from 127.0.98.3:54987 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 09:33:12 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:12 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:12.890194 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:12.885620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:53393)
0926 09:33:12.885752 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:12.885762 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:12.885817 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:12.886013 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:12.886014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:12.886174 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:12.886229 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:12.886231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.886968 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.886969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:12.887581 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:12.887584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.887942 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.887943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:12.887980 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:12.888417 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:12.888422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:12.890136 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260926 09:33:12.890312 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:12.885745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53393 (local address 127.0.98.62:44461)
0926 09:33:12.885872 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:12.885874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:12.885895 (+ 21us) server_negotiation.cc:407] Connection header received
0926 09:33:12.885925 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:12.885926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:12.885936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:12.885994 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:12.886325 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.886883 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.887672 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:12.887868 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:12.887932 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:12.890229 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53393: BlockingRecv error: recv got EOF from 127.0.98.1:53393 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
Sep 26 09:33:13 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:13 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:13.020655 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:13.015728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:59279)
0926 09:33:13.015905 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:13.015917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:13.016017 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:13.016171 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:13.016172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:13.016362 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:13.016413 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:13.016415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.017101 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.017102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:13.017637 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:13.017639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.018020 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.018021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:13.018080 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:13.018585 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:13.018590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:13.020559 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
W20260926 09:33:13.020720 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:13.015819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59279 (local address 127.0.98.62:44461)
0926 09:33:13.015897 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:13.015899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:13.015997 (+ 98us) server_negotiation.cc:407] Connection header received
0926 09:33:13.016076 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:13.016077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:13.016091 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:13.016166 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:13.016501 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.017009 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.017730 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.017931 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.018020 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:13.020657 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59279: BlockingRecv error: recv got EOF from 127.0.98.2:59279 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 09:33:13 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:13 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:13.137795 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:13.133168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:34113)
0926 09:33:13.133332 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:13.133347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:13.133440 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:13.133630 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:13.133631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:13.133846 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:13.133920 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:13.133923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.134618 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.134619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:13.135144 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:13.135146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.135570 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.135571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:13.135611 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:13.136035 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:13.136039 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:13.137737 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260926 09:33:13.137974 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:13.133293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34113 (local address 127.0.98.62:44461)
0926 09:33:13.133361 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:13.133363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:13.133434 (+ 71us) server_negotiation.cc:407] Connection header received
0926 09:33:13.133526 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:13.133527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:13.133542 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:13.133603 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:13.134008 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.134542 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.135231 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.135504 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.135561 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:13.137858 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34113: BlockingRecv error: recv got EOF from 127.0.98.3:34113 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 09:33:13 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:13 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:13.895206 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:13.890732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:47877)
0926 09:33:13.890897 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:13.890907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:13.890956 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:13.891150 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:13.891151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:13.891345 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:13.891405 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:13.891406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.892044 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.892045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:13.892550 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:13.892551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.892861 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.892862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:13.892899 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:13.893316 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:13.893321 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:13.895105 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
W20260926 09:33:13.895310 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:13.890895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47877 (local address 127.0.98.62:44461)
0926 09:33:13.891026 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:13.891027 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:13.891034 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:33:13.891056 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:13.891057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:13.891067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:13.891131 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:13.891473 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.891975 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.892618 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:13.892796 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:13.892845 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:13.895227 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47877: BlockingRecv error: recv got EOF from 127.0.98.1:47877 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
Sep 26 09:33:14 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:14 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:14.026072 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:14.021214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:45483)
0926 09:33:14.021355 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:14.021365 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:14.021423 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:14.021613 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:14.021615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:14.021821 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:14.021874 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:14.021876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.022668 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.022669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:14.023230 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:14.023232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.023588 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.023589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:14.023634 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:14.024102 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:14.024106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:14.026013 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
W20260926 09:33:14.026139 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:14.021295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45483 (local address 127.0.98.62:44461)
0926 09:33:14.021374 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:14.021381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:14.021408 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:33:14.021481 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:14.021482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:14.021493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:14.021551 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:14.021946 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.022570 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.023305 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.023515 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.023568 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:14.026093 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45483: BlockingRecv error: recv got EOF from 127.0.98.2:45483 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 09:33:14 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:14 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:14.142985 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:14.138249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:40287)
0926 09:33:14.138419 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:14.138431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:14.138510 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:14.138675 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:14.138676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:14.138873 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:14.138918 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:14.138919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.139730 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.139731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:14.140259 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:14.140261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.140680 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.140681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:14.140739 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:14.141149 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:14.141153 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:14.142889 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
W20260926 09:33:14.143039 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:14.138324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40287 (local address 127.0.98.62:44461)
0926 09:33:14.138411 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:14.138414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:14.138489 (+ 75us) server_negotiation.cc:407] Connection header received
0926 09:33:14.138565 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:14.138566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:14.138580 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:14.138666 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:14.139002 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.139652 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.140364 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.140571 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.140678 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:14.142972 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40287: BlockingRecv error: recv got EOF from 127.0.98.3:40287 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 09:33:14 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:14 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:14.900276 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:14.895759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:53027)
0926 09:33:14.895922 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:14.895935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:14.895986 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:14.896097 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:14.896098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:14.896256 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:14.896306 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:14.896307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.896923 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.896924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:14.897435 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:14.897437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.897754 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.897755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:14.897794 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:14.898300 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:14.898305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:14.900219 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
W20260926 09:33:14.900411 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:14.895811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53027 (local address 127.0.98.62:44461)
0926 09:33:14.895966 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:14.895968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:14.895989 (+ 21us) server_negotiation.cc:407] Connection header received
0926 09:33:14.896031 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:14.896032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:14.896042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:14.896086 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:14.896360 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.896856 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.897504 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:14.897689 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:14.897740 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:14.900319 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53027: BlockingRecv error: recv got EOF from 127.0.98.1:53027 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
Sep 26 09:33:15 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:15 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:15.038898 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:15.026581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:50095)
0926 09:33:15.026717 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:15.026727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:15.026780 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:15.026913 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:15.026914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:15.027081 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:15.027130 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:15.027132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.027923 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.027924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:15.028435 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:15.028437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.028747 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.028748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:15.028784 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:15.037037 (+ 8253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:15.037043 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:15.038839 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260926 09:33:15.038995 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:15.026667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50095 (local address 127.0.98.62:44461)
0926 09:33:15.026754 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:15.026756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:15.026767 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:33:15.026838 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:15.026838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:15.026849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:15.026893 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:15.027191 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.027855 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.028505 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.028686 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.028733 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:15.038926 (+ 10193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50095: BlockingRecv error: recv got EOF from 127.0.98.2:50095 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 09:33:15 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:15 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:15.148275 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:15.143465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:48127)
0926 09:33:15.143630 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:15.143641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:15.143731 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:15.143888 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:15.143889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:15.144070 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:15.144118 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:15.144120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.144791 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.144792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:15.145331 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:15.145332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.145716 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.145717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:15.145775 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:15.146291 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:15.146297 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:15.148177 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
W20260926 09:33:15.148322 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:15.143543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48127 (local address 127.0.98.62:44461)
0926 09:33:15.143613 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:15.143615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:15.143710 (+ 95us) server_negotiation.cc:407] Connection header received
0926 09:33:15.143793 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:15.143794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:15.143808 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:15.143876 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:15.144200 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.144708 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.145451 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.145644 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.145717 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:15.148254 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48127: BlockingRecv error: recv got EOF from 127.0.98.3:48127 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 09:33:15 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:15 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:15.904937 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:15.900698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:47591)
0926 09:33:15.900838 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:15.900850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:15.900930 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:15.901065 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:15.901066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:15.901224 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:15.901281 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:15.901283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.901926 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.901927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:15.902442 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:15.902443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.902757 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.902758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:15.902795 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:15.903228 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:15.903232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:15.904880 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260926 09:33:15.905042 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:15.900757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47591 (local address 127.0.98.62:44461)
0926 09:33:15.900916 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:15.900919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:15.900935 (+ 16us) server_negotiation.cc:407] Connection header received
0926 09:33:15.900994 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:15.900994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:15.901004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:15.901055 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:15.901344 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.901855 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.902514 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:15.902695 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:15.902744 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:15.904966 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47591: BlockingRecv error: recv got EOF from 127.0.98.1:47591 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
Sep 26 09:33:16 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:16 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:16.044603 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:16.039501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:45137)
0926 09:33:16.039684 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:16.039698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:16.039787 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:16.039973 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:16.039975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:16.040186 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:16.040234 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:16.040235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.040958 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.040959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:16.041627 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:16.041630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.042008 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.042009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:16.042056 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:16.042598 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:16.042604 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:16.044503 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
W20260926 09:33:16.044655 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:16.039609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45137 (local address 127.0.98.62:44461)
0926 09:33:16.039697 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:16.039699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:16.039789 (+ 90us) server_negotiation.cc:407] Connection header received
0926 09:33:16.039869 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:16.039870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:16.039881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:16.039950 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:16.040318 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.040881 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.041738 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.041931 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.041986 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:16.044587 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45137: BlockingRecv error: recv got EOF from 127.0.98.2:45137 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 09:33:16 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:16 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:16.153375 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:16.148759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:37703)
0926 09:33:16.148897 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:16.148907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:16.148983 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:16.149148 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:16.149150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:16.149338 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:16.149384 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:16.149386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.150058 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.150059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:16.150585 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:16.150586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.150967 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.150968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:16.151029 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:16.151556 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:16.151562 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:16.153316 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
W20260926 09:33:16.153510 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:16.148894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37703 (local address 127.0.98.62:44461)
0926 09:33:16.148962 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:16.148964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:16.148974 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:33:16.149064 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:16.149065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:16.149079 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:16.149135 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:16.149464 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.149966 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.150680 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.150879 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.150963 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:16.153425 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37703: BlockingRecv error: recv got EOF from 127.0.98.3:37703 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 09:33:16 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:16 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:16.910472 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:16.905513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:49945)
0926 09:33:16.905684 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:16.905695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:16.905754 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:16.905909 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:16.905909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:16.906074 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:16.906141 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:16.906143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.906785 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.906786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:16.907503 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:16.907506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.907950 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.907950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:16.908004 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:16.908487 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:16.908492 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:16.910353 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
W20260926 09:33:16.910521 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:16.905608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49945 (local address 127.0.98.62:44461)
0926 09:33:16.905736 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:16.905738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:16.905749 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:33:16.905815 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:16.905816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:16.905827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:16.905887 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:16.906206 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.906713 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.907582 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:16.907881 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:16.907936 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:16.910443 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49945: BlockingRecv error: recv got EOF from 127.0.98.1:49945 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":53,"threads_started":1}
Sep 26 09:33:17 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:17 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:17.049808 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:17.045174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:50125)
0926 09:33:17.045340 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:17.045351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:17.045413 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:17.045560 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:17.045561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:17.045725 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:17.045773 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:17.045775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.046409 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.046410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:17.047040 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:17.047043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.047467 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.047468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:17.047529 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:17.047986 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:17.047990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:17.049710 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
W20260926 09:33:17.049885 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:17.045247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50125 (local address 127.0.98.62:44461)
0926 09:33:17.045320 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:17.045322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:17.045415 (+ 93us) server_negotiation.cc:407] Connection header received
0926 09:33:17.045484 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:17.045485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:17.045496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:17.045539 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:17.045835 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.046324 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.047129 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.047393 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.047463 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:17.049801 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50125: BlockingRecv error: recv got EOF from 127.0.98.2:50125 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 09:33:17 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:17 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:17.158542 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:17.153858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:50419)
0926 09:33:17.154029 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:17.154042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:17.154111 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:17.154331 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:17.154333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:17.154505 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:17.154554 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:17.154556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.155278 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.155279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:17.155920 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:17.155922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.156265 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.156265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:17.156304 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:17.156747 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:17.156751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:17.158485 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
W20260926 09:33:17.158675 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:17.153952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50419 (local address 127.0.98.62:44461)
0926 09:33:17.154006 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:17.154008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:17.154160 (+ 152us) server_negotiation.cc:407] Connection header received
0926 09:33:17.154195 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:17.154196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:17.154210 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:17.154276 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:17.154625 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.155192 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.155994 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.156204 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.156258 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:17.158591 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50419: BlockingRecv error: recv got EOF from 127.0.98.3:50419 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 09:33:17 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:17 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:17.915985 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:17.910950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:37647)
0926 09:33:17.911096 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:17.911110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:17.911178 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:17.911401 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:17.911401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:17.911567 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:17.911617 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:17.911618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.912348 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.912349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:17.913094 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:17.913096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.913439 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.913439 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:17.913480 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:17.913935 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:17.913940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:17.915929 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
W20260926 09:33:17.916083 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:17.911018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37647 (local address 127.0.98.62:44461)
0926 09:33:17.911244 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:17.911246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:17.911254 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:33:17.911276 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:17.911277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:17.911288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:17.911386 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:17.911681 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.912203 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.913156 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:17.913369 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:17.913424 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:17.916008 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37647: BlockingRecv error: recv got EOF from 127.0.98.1:37647 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":65,"threads_started":1}
Sep 26 09:33:18 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:18 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:18.055267 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:18.050387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:48327)
0926 09:33:18.050543 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:18.050554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:18.050616 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:18.050761 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:18.050762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:18.050921 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:18.050969 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:18.050970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.051678 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.051679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:18.052184 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:18.052186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.052516 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.052516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:18.052552 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:18.053045 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:18.053050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:18.055149 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
W20260926 09:33:18.055285 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:18.050516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48327 (local address 127.0.98.62:44461)
0926 09:33:18.050584 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:18.050585 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:18.050594 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:33:18.050676 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:18.050677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:18.050687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:18.050743 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:18.051031 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.051614 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.052256 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.052450 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.052502 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:18.055226 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48327: BlockingRecv error: recv got EOF from 127.0.98.2:48327 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 09:33:18 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:18 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:18.163736 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:18.158988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:43263)
0926 09:33:18.159160 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:18.159171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:18.159259 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:18.159445 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:18.159446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:18.159630 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:18.159694 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:18.159696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.160384 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.160385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:18.160930 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:18.160932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.161331 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.161332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:18.161413 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:18.161912 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:18.161924 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:18.163643 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
W20260926 09:33:18.163776 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:18.159080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43263 (local address 127.0.98.62:44461)
0926 09:33:18.159150 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:18.159153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:18.159242 (+ 89us) server_negotiation.cc:407] Connection header received
0926 09:33:18.159352 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:18.159353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:18.159368 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:18.159438 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:18.159779 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.160287 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.161015 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.161255 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.161330 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:18.163713 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43263: BlockingRecv error: recv got EOF from 127.0.98.3:43263 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 09:33:18 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:18 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:18.921485 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:18.916535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:49929)
0926 09:33:18.916690 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:18.916704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:18.916773 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:18.916954 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:18.916955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:18.917125 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:18.917177 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:18.917179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.917877 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.917877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:18.918403 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:18.918405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.918919 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.918920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:18.918992 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:18.919589 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:18.919594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:18.921426 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
W20260926 09:33:18.921625 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:18.916591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49929 (local address 127.0.98.62:44461)
0926 09:33:18.916749 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:18.916751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:18.916761 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:33:18.916863 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:18.916864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:18.916875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:18.916934 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:18.917287 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.917798 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.918492 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:18.918827 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:18.918921 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:18.921532 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49929: BlockingRecv error: recv got EOF from 127.0.98.1:49929 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
Sep 26 09:33:19 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:19 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:19.060360 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:19.055791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:46047)
0926 09:33:19.055958 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:19.055973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:19.056044 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:19.056169 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:19.056169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:19.056320 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:19.056371 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:19.056373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.056984 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.056984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:19.057497 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:19.057499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.057810 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.057811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:19.057856 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:19.058292 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:19.058296 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:19.060259 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
W20260926 09:33:19.060395 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:19.055924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46047 (local address 127.0.98.62:44461)
0926 09:33:19.055984 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:19.055986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:19.056040 (+ 54us) server_negotiation.cc:407] Connection header received
0926 09:33:19.056088 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:19.056089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:19.056099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:19.056148 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:19.056433 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.056919 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.057574 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.057748 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.057810 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:19.060335 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46047: BlockingRecv error: recv got EOF from 127.0.98.2:46047 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 09:33:19 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:19 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:19.168910 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:19.164255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:42877)
0926 09:33:19.164411 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:19.164430 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:19.164486 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:19.164622 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:19.164623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:19.164781 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:19.164833 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:19.164834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.165443 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.165444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:19.165956 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:19.165957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.166352 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.166353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:19.166415 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:19.166981 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:19.166986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:19.168826 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
W20260926 09:33:19.168969 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:19.164306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42877 (local address 127.0.98.62:44461)
0926 09:33:19.164375 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:19.164382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:19.164487 (+ 105us) server_negotiation.cc:407] Connection header received
0926 09:33:19.164546 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:19.164547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:19.164560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:19.164603 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:19.164892 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.165374 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.166072 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.166273 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.166352 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:19.168920 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42877: BlockingRecv error: recv got EOF from 127.0.98.3:42877 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 09:33:19 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:19 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:19.926471 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:19.921979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:33805)
0926 09:33:19.922133 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:19.922146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:19.922222 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:19.922364 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:19.922365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:19.922525 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:19.922575 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:19.922576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.923242 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.923244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:19.923842 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:19.923844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.924188 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.924189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:19.924233 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:19.924672 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:19.924677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:19.926405 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
W20260926 09:33:19.926573 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:19.922078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33805 (local address 127.0.98.62:44461)
0926 09:33:19.922208 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:19.922210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:19.922218 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:33:19.922280 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:19.922281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:19.922291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:19.922342 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:19.922641 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.923145 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.923924 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:19.924110 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:19.924165 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:19.926494 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33805: BlockingRecv error: recv got EOF from 127.0.98.1:33805 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
Sep 26 09:33:20 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:20 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:20.069732 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:20.060881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:40931)
0926 09:33:20.061037 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:20.061048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:20.061104 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:20.061212 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:20.061213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:20.061378 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:20.061426 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:20.061428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.062020 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.062020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:20.062540 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:20.062542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.062876 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.062876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:20.062912 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:20.067830 (+ 4918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:20.067840 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:20.069657 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
W20260926 09:33:20.069854 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:20.060897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40931 (local address 127.0.98.62:44461)
0926 09:33:20.060966 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:20.060969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:20.061108 (+ 139us) server_negotiation.cc:407] Connection header received
0926 09:33:20.061148 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:20.061148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:20.061159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:20.061203 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:20.061482 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.061955 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.062622 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.062809 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.062869 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:20.069777 (+ 6908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40931: BlockingRecv error: recv got EOF from 127.0.98.2:40931 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 09:33:20 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:20 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:20.174093 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:20.169381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.3:48189)
0926 09:33:20.169565 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:20.169577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:20.169646 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:20.169797 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:20.169798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:20.169992 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:20.170039 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:20.170041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.170723 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.170725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:20.171271 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:20.171273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.171715 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.171716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:20.171774 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:20.172240 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:20.172244 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:20.173997 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
W20260926 09:33:20.174147 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:20.169478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48189 (local address 127.0.98.62:44461)
0926 09:33:20.169549 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:20.169552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:20.169643 (+ 91us) server_negotiation.cc:407] Connection header received
0926 09:33:20.169702 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:20.169703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:20.169718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:20.169791 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:20.170121 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.170625 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.171433 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.171627 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.171716 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:20.174080 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48189: BlockingRecv error: recv got EOF from 127.0.98.3:48189 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 09:33:20 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:20 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:20.931511 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:20.927015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.1:52943)
0926 09:33:20.927170 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:20.927180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:20.927228 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:20.927383 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:20.927384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:20.927547 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:20.927595 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:20.927596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.928237 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.928237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:20.928776 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:20.928777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.929102 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.929103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:20.929140 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:20.929560 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:20.929565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:20.931403 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
W20260926 09:33:20.931559 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:20.927078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52943 (local address 127.0.98.62:44461)
0926 09:33:20.927222 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:20.927224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:20.927262 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:33:20.927281 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:20.927282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:20.927292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:20.927360 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:20.927661 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.928149 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.928848 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:20.929038 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:20.929091 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:20.931485 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52943: BlockingRecv error: recv got EOF from 127.0.98.1:52943 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
Sep 26 09:33:21 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:33:21 dist-test-slave-21h8 krb5kdc[8519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:33:21.074810 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.070285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44461 (local address 127.0.98.2:37647)
0926 09:33:21.070424 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:21.070435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:21.070489 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:21.070621 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:21.070622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:21.070778 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:21.070828 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:21.070829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.071542 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.071542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:21.072068 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:21.072070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.072410 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.072411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:21.072449 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.072921 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:21.072926 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:21.074731 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44461: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
W20260926 09:33:21.074931 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:21.070343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37647 (local address 127.0.98.62:44461)
0926 09:33:21.070408 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:21.070410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:21.070490 (+ 80us) server_negotiation.cc:407] Connection header received
0926 09:33:21.070535 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:21.070536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:21.070547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:21.070605 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:21.070931 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.071470 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.072150 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.072340 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.072398 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.074843 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37647: BlockingRecv error: recv got EOF from 127.0.98.2:37647 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:33:21.086941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 9102
I20260926 09:33:21.091758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 9254
I20260926 09:33:21.095891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 9387
I20260926 09:33:21.100121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 9032
2026-09-26T09:33:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17500 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T09:33:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:33:21Z Disabled control of system clock
I20260926 09:33:21.126950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36969
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 09:33:21.207589 9583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:21.207785 9583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:21.207804 9583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:21.208920 9583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:33:21.208952 9583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:21.208966 9583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:21.208977 9583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:33:21.208989 9583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:33:21.210223 9583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:21.210417 9583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:21.210708 9583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:21.215516 9588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.215576 9583 server_base.cc:1034] running on GCE node
W20260926 09:33:21.215517 9589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:21.215505 9591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.215987 9583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:21.216295 9583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:21.217443 9583 hybrid_clock.cc:648] HybridClock initialized: now 1790415201217435 us; error 33 us; skew 500 ppm
I20260926 09:33:21.218621 9583 webserver.cc:460] Webserver started at http://127.0.98.62:41103/ using document root <none> and password file <none>
I20260926 09:33:21.218886 9583 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:21.218935 9583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:21.219103 9583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:21.220232 9583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "452de03b532248469114b390b1daac8b"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "5caa599822c5c96b8e65681e2b5f8818"
server_key_iv: "1b87dfd20ceaca982fe292c9afd5db94"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.220630 9583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "452de03b532248469114b390b1daac8b"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "5caa599822c5c96b8e65681e2b5f8818"
server_key_iv: "1b87dfd20ceaca982fe292c9afd5db94"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.221931 9583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:33:21.222721 9597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.222908 9583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:21.222998 9583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "452de03b532248469114b390b1daac8b"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "5caa599822c5c96b8e65681e2b5f8818"
server_key_iv: "1b87dfd20ceaca982fe292c9afd5db94"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.223085 9583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:21.245085 9583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:21.245438 9583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:21.245600 9583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:21.249694 9583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36969
I20260926 09:33:21.249727 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36969 every 8 connection(s)
I20260926 09:33:21.250075 9583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:33:21.250798 9650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:21.251121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 9583
I20260926 09:33:21.251210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:33:21.251464 392 external_mini_cluster.cc:1428] Setting key 768073b208efe341a44f42340175a232
I20260926 09:33:21.254637 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b: Bootstrap starting.
I20260926 09:33:21.255409 9650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:21.255688 9650 log.cc:824] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:21.256502 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b: No bootstrap required, opened a new log
I20260926 09:33:21.257771 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.251999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.0.98.62:36969)
0926 09:33:21.252197 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:21.252206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:21.252812 (+ 606us) server_negotiation.cc:407] Connection header received
0926 09:33:21.253259 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:21.253265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:21.253327 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:21.253443 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:21.253689 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.254548 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.255562 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.256129 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.256250 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.257002 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:21.257006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:21.257007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:33:21.257245 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:21.257294 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:33:21.257306 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:21.257397 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:21.257476 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 09:33:21.257559 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260926 09:33:21.259368 9650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452de03b532248469114b390b1daac8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260926 09:33:21.259560 9650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:21.259646 9650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452de03b532248469114b390b1daac8b, State: Initialized, Role: FOLLOWER
I20260926 09:33:21.259802 9650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452de03b532248469114b390b1daac8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260926 09:33:21.259917 9650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:33:21.259963 9650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:33:21.260013 9650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:21.260543 9650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452de03b532248469114b390b1daac8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260926 09:33:21.260663 9650 leader_election.cc:302] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452de03b532248469114b390b1daac8b; no voters:
I20260926 09:33:21.260834 9650 leader_election.cc:288] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:33:21.260872 9655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:21.261057 9655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [term 1 LEADER]: Becoming Leader. State: Replica: 452de03b532248469114b390b1daac8b, State: Running, Role: LEADER
I20260926 09:33:21.261091 9650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:33:21.261173 9655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452de03b532248469114b390b1daac8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260926 09:33:21.261580 9657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452de03b532248469114b390b1daac8b. Latest consensus state: current_term: 1 leader_uuid: "452de03b532248469114b390b1daac8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452de03b532248469114b390b1daac8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } }
I20260926 09:33:21.261711 9657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:21.261579 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452de03b532248469114b390b1daac8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452de03b532248469114b390b1daac8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } }
I20260926 09:33:21.261927 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:21.262014 9664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:33:21.262553 9664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:33:21.264164 9664 catalog_manager.cc:1347] Generated new cluster ID: 57265b32f7124fbbb72d04c7ba7434a0
I20260926 09:33:21.264217 9664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:33:21.282667 9664 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:33:21.283179 9664 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:33:21.287672 9664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452de03b532248469114b390b1daac8b: Generated new TSK 0
I20260926 09:33:21.287878 9664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:33:21.290261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--builtin_ntp_servers=127.0.98.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 09:33:21.377478 9674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:21.377694 9674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:21.377717 9674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:21.378906 9674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:21.378947 9674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:21.378973 9674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:33:21.380393 9674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:21.380633 9674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:21.380955 9674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:21.386933 9679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:21.386973 9680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:21.386966 9682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.387092 9674 server_base.cc:1034] running on GCE node
I20260926 09:33:21.387449 9674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:21.387745 9674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:21.388906 9674 hybrid_clock.cc:648] HybridClock initialized: now 1790415201388891 us; error 33 us; skew 500 ppm
I20260926 09:33:21.390345 9674 webserver.cc:460] Webserver started at http://127.0.98.1:33389/ using document root <none> and password file <none>
I20260926 09:33:21.390596 9674 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:21.390666 9674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:21.390802 9674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:21.391984 9674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "401fb62fbf7545f18bd1d9d9b0a02406"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "a55aabd0115c842ada0d78826d36add7"
server_key_iv: "788806886202156cd2cdb839f8f56d85"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.392364 9674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "401fb62fbf7545f18bd1d9d9b0a02406"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "a55aabd0115c842ada0d78826d36add7"
server_key_iv: "788806886202156cd2cdb839f8f56d85"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.393688 9674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:33:21.394513 9688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.394757 9674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:21.394855 9674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "401fb62fbf7545f18bd1d9d9b0a02406"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "a55aabd0115c842ada0d78826d36add7"
server_key_iv: "788806886202156cd2cdb839f8f56d85"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.394927 9674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:21.408507 9674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:21.408798 9674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:21.408901 9674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:21.409085 9674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:21.409462 9674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:21.409502 9674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.409548 9674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:21.409577 9674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.415419 9674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42989
I20260926 09:33:21.415479 9801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42989 every 8 connection(s)
I20260926 09:33:21.415839 9674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:33:21.421036 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.416613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46107 (local address 127.0.98.62:36969)
0926 09:33:21.416679 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:21.416681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:21.416961 (+ 280us) server_negotiation.cc:407] Connection header received
0926 09:33:21.417453 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:21.417454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:21.417467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:21.417525 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:21.418183 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.418684 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.419714 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.419913 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.419972 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.420784 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:21.420786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:21.420786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:33:21.420790 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:21.420803 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:33:21.420806 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:21.420862 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:21.420952 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 09:33:21.420976 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:33:21.421180 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.416614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.1:46107)
0926 09:33:21.416834 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:21.416869 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:21.417366 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:21.417593 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:21.417596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:21.417605 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:21.418097 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:21.418101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.418778 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.418779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:21.419637 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:21.419639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.419995 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.419996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:21.420103 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.420646 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:33:21.420646 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:21.420675 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:33:21.420698 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:21.420879 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:21.420883 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:21.420885 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:33:21.420939 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:33:21.421020 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260926 09:33:21.421729 9802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260926 09:33:21.421864 9802 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:21.422049 9802 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260926 09:33:21.422570 9614 ts_manager.cc:194] Registered new tserver with Master: 401fb62fbf7545f18bd1d9d9b0a02406 (127.0.98.1:42989)
I20260926 09:33:21.423190 9614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46107
I20260926 09:33:21.425545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 9674
I20260926 09:33:21.425637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:33:21.425861 392 external_mini_cluster.cc:1428] Setting key 8f7081fa3b76ae00f02752a8471c87fd
I20260926 09:33:21.426754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--builtin_ntp_servers=127.0.98.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 09:33:21.507515 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:21.507735 9805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:21.507768 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:21.508888 9805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:21.508949 9805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:21.508988 9805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:33:21.510299 9805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:21.510560 9805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:21.510845 9805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:21.516083 9813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.516252 9805 server_base.cc:1034] running on GCE node
W20260926 09:33:21.516094 9811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:21.516094 9810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.516522 9805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:21.516777 9805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:21.517953 9805 hybrid_clock.cc:648] HybridClock initialized: now 1790415201517930 us; error 43 us; skew 500 ppm
I20260926 09:33:21.519225 9805 webserver.cc:460] Webserver started at http://127.0.98.2:36427/ using document root <none> and password file <none>
I20260926 09:33:21.519485 9805 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:21.519555 9805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:21.519693 9805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:21.520777 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "688045764ba64df7a20f0ace071f99cf"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "3ae378a8b47a4c4d5d601af98d8b644f"
server_key_iv: "d979f7a2df067b172cac189acd27d840"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.521144 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "688045764ba64df7a20f0ace071f99cf"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "3ae378a8b47a4c4d5d601af98d8b644f"
server_key_iv: "d979f7a2df067b172cac189acd27d840"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.522557 9805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:33:21.523450 9819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.523630 9805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:21.523728 9805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "688045764ba64df7a20f0ace071f99cf"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "3ae378a8b47a4c4d5d601af98d8b644f"
server_key_iv: "d979f7a2df067b172cac189acd27d840"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.523818 9805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:21.532548 9805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:21.532831 9805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:21.532987 9805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:21.533200 9805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:21.533527 9805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:21.533581 9805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.533615 9805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:21.533643 9805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.539721 9805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41799
I20260926 09:33:21.539767 9932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41799 every 8 connection(s)
I20260926 09:33:21.540088 9805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:33:21.541575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 9805
I20260926 09:33:21.541704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:33:21.541909 392 external_mini_cluster.cc:1428] Setting key 10c952829e506667774a30d3a7a14e65
I20260926 09:33:21.542886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--builtin_ntp_servers=127.0.98.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 09:33:21.546245 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.540796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38719 (local address 127.0.98.62:36969)
0926 09:33:21.540843 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:21.540846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:21.541085 (+ 239us) server_negotiation.cc:407] Connection header received
0926 09:33:21.541789 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:21.541791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:21.541807 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:21.541883 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:21.542767 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.543401 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.544720 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.544941 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.545007 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.545890 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:21.545892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:21.545892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:33:21.545897 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:21.545914 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:33:21.545919 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:21.545979 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:21.546092 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 09:33:21.546122 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:33:21.546370 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.540753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.2:38719)
0926 09:33:21.540921 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:21.540974 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:21.541631 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:21.541959 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:21.541963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:21.541974 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:21.542677 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:21.542684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.543493 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.543494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:21.544612 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:21.544615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.545032 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.545033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:21.545191 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.545751 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:33:21.545752 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:21.545781 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:33:21.545801 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:21.545996 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:21.546002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:21.546005 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:33:21.546085 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:33:21.546164 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260926 09:33:21.546955 9933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260926 09:33:21.547080 9933 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:21.547263 9933 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260926 09:33:21.547703 9614 ts_manager.cc:194] Registered new tserver with Master: 688045764ba64df7a20f0ace071f99cf (127.0.98.2:41799)
I20260926 09:33:21.548072 9614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38719
W20260926 09:33:21.624387 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:21.624642 9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:21.624676 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:21.625871 9936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:21.625926 9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:21.625967 9936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:33:21.627650 9936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:21.627911 9936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:21.628216 9936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:21.633195 9941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:21.633176 9942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.633324 9936 server_base.cc:1034] running on GCE node
W20260926 09:33:21.633178 9944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.633600 9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:21.633831 9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:21.634994 9936 hybrid_clock.cc:648] HybridClock initialized: now 1790415201634985 us; error 46 us; skew 500 ppm
I20260926 09:33:21.636265 9936 webserver.cc:460] Webserver started at http://127.0.98.3:37375/ using document root <none> and password file <none>
I20260926 09:33:21.636579 9936 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:21.636647 9936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:21.636790 9936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:21.637890 9936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bf57622db7b44e093a80fed0c8196bf"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "6a29368961a0e3408ad7d14a6391c25c"
server_key_iv: "6d15e0eb2e279cfbeabb1274d2658019"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.638262 9936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bf57622db7b44e093a80fed0c8196bf"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "6a29368961a0e3408ad7d14a6391c25c"
server_key_iv: "6d15e0eb2e279cfbeabb1274d2658019"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.639600 9936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:33:21.640295 9950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.640497 9936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:33:21.640590 9936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "8bf57622db7b44e093a80fed0c8196bf"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "6a29368961a0e3408ad7d14a6391c25c"
server_key_iv: "6d15e0eb2e279cfbeabb1274d2658019"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.640674 9936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:21.651542 9936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:21.651839 9936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:21.651983 9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:21.652200 9936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:21.652509 9936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:21.652559 9936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.652601 9936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:21.652627 9936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.658465 9936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41677
I20260926 09:33:21.658509 10063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41677 every 8 connection(s)
I20260926 09:33:21.658815 9936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:33:21.664001 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.659516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55793 (local address 127.0.98.62:36969)
0926 09:33:21.659579 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:21.659581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:21.659859 (+ 278us) server_negotiation.cc:407] Connection header received
0926 09:33:21.660336 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:21.660337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:21.660346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:21.660399 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:21.661122 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.661615 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.662688 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.662870 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.662924 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.663778 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:21.663779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:21.663779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:33:21.663783 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:21.663796 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:33:21.663800 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:21.663843 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:21.663922 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 09:33:21.663942 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:33:21.664247 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.659548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.3:55793)
0926 09:33:21.659741 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:21.659774 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:21.660256 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:21.660456 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:21.660459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:21.660468 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:21.661002 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:21.661006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.661706 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.661707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:21.662616 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:21.662618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:21.662942 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:21.662942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:21.663045 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:21.663656 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:33:21.663657 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:21.663685 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:33:21.663707 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:21.663858 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:21.663862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:21.663864 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:33:21.663915 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:33:21.663993 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260926 09:33:21.664757 10064 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260926 09:33:21.664861 10064 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:21.665063 10064 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260926 09:33:21.665459 9614 ts_manager.cc:194] Registered new tserver with Master: 8bf57622db7b44e093a80fed0c8196bf (127.0.98.3:41677)
I20260926 09:33:21.665832 9614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55793
I20260926 09:33:21.667569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 9936
I20260926 09:33:21.667640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:33:21.667769 392 external_mini_cluster.cc:1428] Setting key 40031ca34b8ac96aa0fdfb6049bbe876
I20260926 09:33:21.668619 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 09:33:21.670367 9653 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:21.669919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.0.98.62:36969)
0926 09:33:21.669960 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:21.669963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:21.670074 (+ 111us) server_negotiation.cc:407] Connection header received
0926 09:33:21.670110 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:21.670111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:21.670118 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:21.670168 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:21.670312 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45720: BlockingRecv error: recv got EOF from 127.0.0.1:45720 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:33:21.670459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 9674
I20260926 09:33:21.675240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 9805
I20260926 09:33:21.679509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 9936
I20260926 09:33:21.683490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 9583
2026-09-26T09:33:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (582 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T09:33:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:33:21Z Disabled control of system clock
I20260926 09:33:21.709889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35653
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:33:21.788378 10075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:21.788586 10075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:21.788607 10075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:21.789803 10075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:33:21.789855 10075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:21.789870 10075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:21.789881 10075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:33:21.789893 10075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:33:21.791370 10075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:21.791621 10075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:21.791913 10075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:21.796716 10081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:21.796718 10083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.796743 10075 server_base.cc:1034] running on GCE node
W20260926 09:33:21.796749 10080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.797149 10075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:21.797430 10075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:21.798569 10075 hybrid_clock.cc:648] HybridClock initialized: now 1790415201798558 us; error 26 us; skew 500 ppm
I20260926 09:33:21.800088 10075 webserver.cc:460] Webserver started at http://127.0.98.62:40859/ using document root <none> and password file <none>
I20260926 09:33:21.800287 10075 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:21.800328 10075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:21.800441 10075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:21.801443 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "4c9f4ebb71c648d6ac6fd00213cc868c"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "851e2f74953dfdc09d2a12ef486a3d17"
server_key_iv: "9203da47983827908d83b33eba7d198b"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.801741 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c9f4ebb71c648d6ac6fd00213cc868c"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "851e2f74953dfdc09d2a12ef486a3d17"
server_key_iv: "9203da47983827908d83b33eba7d198b"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.803041 10075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:33:21.803810 10089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.803990 10075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.804078 10075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "4c9f4ebb71c648d6ac6fd00213cc868c"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "851e2f74953dfdc09d2a12ef486a3d17"
server_key_iv: "9203da47983827908d83b33eba7d198b"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.804131 10075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:21.807300 10075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:21.807574 10075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:21.807663 10075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:21.811228 10075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35653
I20260926 09:33:21.811285 10141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35653 every 8 connection(s)
I20260926 09:33:21.811606 10075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:33:21.812232 10142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:21.813894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 10075
I20260926 09:33:21.813987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:33:21.814162 392 external_mini_cluster.cc:1428] Setting key af34055ebf17d7eab70038c56240173d
I20260926 09:33:21.815541 10142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c: Bootstrap starting.
I20260926 09:33:21.816267 10142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:21.816610 10142 log.cc:824] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:21.816960 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.814693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.0.98.62:35653)
0926 09:33:21.814954 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:21.814963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:21.814998 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:33:21.815579 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:21.815588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:21.815641 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:21.815791 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:21.816494 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:21.816497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:21.816498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:33:21.816502 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:21.816557 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:33:21.816572 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:21.816673 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:21.816721 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 09:33:21.816784 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":67,"threads_started":1}
I20260926 09:33:21.817574 10142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c: No bootstrap required, opened a new log
I20260926 09:33:21.819429 10142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f4ebb71c648d6ac6fd00213cc868c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35653 } }
I20260926 09:33:21.819576 10142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:21.819607 10142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9f4ebb71c648d6ac6fd00213cc868c, State: Initialized, Role: FOLLOWER
I20260926 09:33:21.819741 10142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f4ebb71c648d6ac6fd00213cc868c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35653 } }
I20260926 09:33:21.819829 10142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:33:21.819854 10142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:33:21.819885 10142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:21.820402 10142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f4ebb71c648d6ac6fd00213cc868c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35653 } }
I20260926 09:33:21.820489 10142 leader_election.cc:302] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c9f4ebb71c648d6ac6fd00213cc868c; no voters:
I20260926 09:33:21.820631 10142 leader_election.cc:288] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:33:21.820694 10147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:21.820860 10147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [term 1 LEADER]: Becoming Leader. State: Replica: 4c9f4ebb71c648d6ac6fd00213cc868c, State: Running, Role: LEADER
I20260926 09:33:21.820888 10142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:33:21.820994 10147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f4ebb71c648d6ac6fd00213cc868c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35653 } }
I20260926 09:33:21.821388 10149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9f4ebb71c648d6ac6fd00213cc868c. Latest consensus state: current_term: 1 leader_uuid: "4c9f4ebb71c648d6ac6fd00213cc868c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f4ebb71c648d6ac6fd00213cc868c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35653 } } }
I20260926 09:33:21.821372 10148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c9f4ebb71c648d6ac6fd00213cc868c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f4ebb71c648d6ac6fd00213cc868c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35653 } } }
I20260926 09:33:21.821482 10149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:21.821488 10148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:21.821687 10154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:33:21.822165 10154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:33:21.823766 10154 catalog_manager.cc:1347] Generated new cluster ID: 3004e3c00d234fcdb5faf0d42878a31a
I20260926 09:33:21.823819 10154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:33:21.832062 10154 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:33:21.832231 10154 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:33:21.835677 10154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c9f4ebb71c648d6ac6fd00213cc868c: Generated new TSK 0
I20260926 09:33:21.835881 10154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:33:21.842192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35653
--builtin_ntp_servers=127.0.98.20:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:33:21.922412 10166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:21.922674 10166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:21.922727 10166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:21.923938 10166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:21.923976 10166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:21.924002 10166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:33:21.925369 10166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:21.925617 10166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:21.925936 10166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:21.930948 10171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:21.930926 10174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:21.930926 10172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:21.931530 10166 server_base.cc:1034] running on GCE node
I20260926 09:33:21.931768 10166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:21.932025 10166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:21.933261 10166 hybrid_clock.cc:648] HybridClock initialized: now 1790415201933245 us; error 37 us; skew 500 ppm
I20260926 09:33:21.934808 10166 webserver.cc:460] Webserver started at http://127.0.98.1:41575/ using document root <none> and password file <none>
I20260926 09:33:21.935078 10166 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:21.935151 10166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:21.935303 10166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:21.936416 10166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "f6fc9c1f52854188bed438d75488e531"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "403a79a9520ed70e287ba8fb14dd4eaa"
server_key_iv: "7becadcf4ebeb0872e4893197912ec3c"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.936776 10166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f6fc9c1f52854188bed438d75488e531"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "403a79a9520ed70e287ba8fb14dd4eaa"
server_key_iv: "7becadcf4ebeb0872e4893197912ec3c"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.938287 10166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:33:21.939116 10180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.939409 10166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:33:21.939527 10166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "f6fc9c1f52854188bed438d75488e531"
format_stamp: "Formatted at 2026-09-26 09:33:21 on dist-test-slave-21h8"
server_key: "403a79a9520ed70e287ba8fb14dd4eaa"
server_key_iv: "7becadcf4ebeb0872e4893197912ec3c"
server_key_version: "encryptionkey@0"
I20260926 09:33:21.939657 10166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:21.943430 10166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:21.943732 10166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:21.943897 10166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:21.944125 10166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:21.944460 10166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:21.944520 10166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.944564 10166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:21.944597 10166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:21.951224 10166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43953
I20260926 09:33:21.951262 10293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43953 every 8 connection(s)
I20260926 09:33:21.951706 10166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:33:21.954875 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.952472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33699 (local address 127.0.98.62:35653)
0926 09:33:21.952557 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:21.952559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:21.952862 (+ 303us) server_negotiation.cc:407] Connection header received
0926 09:33:21.953336 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:21.953337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:21.953346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:21.953399 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:21.954678 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:21.954679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:21.954679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:33:21.954682 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:21.954691 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:33:21.954694 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:21.954732 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:21.954805 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 09:33:21.954817 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 09:33:21.955106 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:21.952458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35653 (local address 127.0.98.1:33699)
0926 09:33:21.952726 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:21.952763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:21.953250 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:21.953456 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:21.953460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:21.953469 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:21.954522 (+ 1053us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:33:21.954523 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:21.954554 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:33:21.954584 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:21.954739 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:21.954742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:21.954744 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:33:21.954799 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:33:21.954851 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20260926 09:33:21.955693 10294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35653
I20260926 09:33:21.955785 10294 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:21.955907 10294 heartbeater.cc:511] Master 127.0.98.62:35653 requested a full tablet report, sending...
I20260926 09:33:21.956409 10106 ts_manager.cc:194] Registered new tserver with Master: f6fc9c1f52854188bed438d75488e531 (127.0.98.1:43953)
I20260926 09:33:21.957274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 10166
I20260926 09:33:21.957345 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:33:21.957557 392 external_mini_cluster.cc:1428] Setting key 6a1053837824fd24025182d13ef76480
I20260926 09:33:21.958523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35653
--builtin_ntp_servers=127.0.98.20:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:33:22.058018 10297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:22.058322 10297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:22.058367 10297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:22.059566 10297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:22.059604 10297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:22.059630 10297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:33:22.060932 10297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:22.061117 10297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:22.061360 10297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:22.066366 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:22.066393 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:22.066383 10305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:22.066622 10297 server_base.cc:1034] running on GCE node
I20260926 09:33:22.066815 10297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:22.067051 10297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:22.068218 10297 hybrid_clock.cc:648] HybridClock initialized: now 1790415202068210 us; error 43 us; skew 500 ppm
I20260926 09:33:22.069429 10297 webserver.cc:460] Webserver started at http://127.0.98.2:38941/ using document root <none> and password file <none>
I20260926 09:33:22.069674 10297 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:22.069741 10297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:22.069891 10297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:22.070971 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "9042e1637c6341d3a102296b868ccef1"
format_stamp: "Formatted at 2026-09-26 09:33:22 on dist-test-slave-21h8"
server_key: "f209b873d4a04f9f6f7075318f2281d0"
server_key_iv: "3a37654533735ab324cc94df586d3bb4"
server_key_version: "encryptionkey@0"
I20260926 09:33:22.071396 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9042e1637c6341d3a102296b868ccef1"
format_stamp: "Formatted at 2026-09-26 09:33:22 on dist-test-slave-21h8"
server_key: "f209b873d4a04f9f6f7075318f2281d0"
server_key_iv: "3a37654533735ab324cc94df586d3bb4"
server_key_version: "encryptionkey@0"
I20260926 09:33:22.072679 10297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:33:22.073410 10311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:22.073585 10297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:33:22.073664 10297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "9042e1637c6341d3a102296b868ccef1"
format_stamp: "Formatted at 2026-09-26 09:33:22 on dist-test-slave-21h8"
server_key: "f209b873d4a04f9f6f7075318f2281d0"
server_key_iv: "3a37654533735ab324cc94df586d3bb4"
server_key_version: "encryptionkey@0"
I20260926 09:33:22.073716 10297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:22.076892 10297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:22.077099 10297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:22.077188 10297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:22.077337 10297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:22.077632 10297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:22.077692 10297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:22.077750 10297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:22.077780 10297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:22.084416 10297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45757
I20260926 09:33:22.084482 10424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45757 every 8 connection(s)
I20260926 09:33:22.084792 10297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:33:22.087559 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:22.085493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56939 (local address 127.0.98.62:35653)
0926 09:33:22.085576 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:22.085578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:22.085769 (+ 191us) server_negotiation.cc:407] Connection header received
0926 09:33:22.086208 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:22.086209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:22.086215 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:22.086267 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:22.087292 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:22.087293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:22.087293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:33:22.087297 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:22.087356 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:33:22.087362 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:22.087417 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:22.087492 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:33:22.087505 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 09:33:22.087751 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:22.085459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35653 (local address 127.0.98.2:56939)
0926 09:33:22.085651 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:22.085684 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:22.086123 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:22.086324 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:22.086327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:22.086335 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:22.087159 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:33:22.087159 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:22.087185 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:33:22.087212 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:22.087416 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:22.087419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:22.087422 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:33:22.087487 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 09:33:22.087533 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260926 09:33:22.088249 10425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35653
I20260926 09:33:22.088340 10425 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:22.088505 10425 heartbeater.cc:511] Master 127.0.98.62:35653 requested a full tablet report, sending...
I20260926 09:33:22.088869 10106 ts_manager.cc:194] Registered new tserver with Master: 9042e1637c6341d3a102296b868ccef1 (127.0.98.2:45757)
I20260926 09:33:22.093883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 10297
I20260926 09:33:22.093982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:33:22.094164 392 external_mini_cluster.cc:1428] Setting key d8239259fe8a65b5455a5f1ba508abfa
I20260926 09:33:22.095096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35653
--builtin_ntp_servers=127.0.98.20:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:33:22.174500 10428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:22.174708 10428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:22.174731 10428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:22.175894 10428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:22.175957 10428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:22.176000 10428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:33:22.177459 10428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:22.177691 10428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:22.177965 10428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:22.182927 10433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:22.182998 10436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:22.182930 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:22.183044 10428 server_base.cc:1034] running on GCE node
I20260926 09:33:22.183362 10428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:22.183568 10428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:22.184742 10428 hybrid_clock.cc:648] HybridClock initialized: now 1790415202184729 us; error 29 us; skew 500 ppm
I20260926 09:33:22.185899 10428 webserver.cc:460] Webserver started at http://127.0.98.3:45049/ using document root <none> and password file <none>
I20260926 09:33:22.186123 10428 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:22.186184 10428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:22.186347 10428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:22.187472 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "85ed033469d745528e8c4de4f7417b88"
format_stamp: "Formatted at 2026-09-26 09:33:22 on dist-test-slave-21h8"
server_key: "14738cb725fbc13d7db632a89d85b1ac"
server_key_iv: "d98dda5b24a3d794757cb93a70441fe9"
server_key_version: "encryptionkey@0"
I20260926 09:33:22.187815 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "85ed033469d745528e8c4de4f7417b88"
format_stamp: "Formatted at 2026-09-26 09:33:22 on dist-test-slave-21h8"
server_key: "14738cb725fbc13d7db632a89d85b1ac"
server_key_iv: "d98dda5b24a3d794757cb93a70441fe9"
server_key_version: "encryptionkey@0"
I20260926 09:33:22.189105 10428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:33:22.189828 10442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:22.190016 10428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:33:22.190109 10428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "85ed033469d745528e8c4de4f7417b88"
format_stamp: "Formatted at 2026-09-26 09:33:22 on dist-test-slave-21h8"
server_key: "14738cb725fbc13d7db632a89d85b1ac"
server_key_iv: "d98dda5b24a3d794757cb93a70441fe9"
server_key_version: "encryptionkey@0"
I20260926 09:33:22.190188 10428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:22.193537 10428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:22.193794 10428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:22.193921 10428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:22.194114 10428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:22.194403 10428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:22.194434 10428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:22.194483 10428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:22.194511 10428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:22.200328 10428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34849
I20260926 09:33:22.200423 10555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34849 every 8 connection(s)
I20260926 09:33:22.200723 10428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:33:22.203572 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:22.201407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35507 (local address 127.0.98.62:35653)
0926 09:33:22.201489 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:22.201491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:22.201673 (+ 182us) server_negotiation.cc:407] Connection header received
0926 09:33:22.202154 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:22.202155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:22.202163 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:22.202252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:22.203362 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:22.203363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:22.203363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:33:22.203368 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:22.203391 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:33:22.203395 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:22.203444 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:22.203508 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 09:33:22.203521 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 09:33:22.203756 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:22.201378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35653 (local address 127.0.98.3:35507)
0926 09:33:22.201555 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:22.201594 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:22.202046 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:22.202309 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:22.202312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:22.202319 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:22.203174 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:33:22.203174 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:22.203200 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:33:22.203225 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:22.203456 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:22.203459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:22.203461 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:33:22.203513 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:33:22.203558 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260926 09:33:22.204226 10556 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35653
I20260926 09:33:22.204344 10556 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:22.204499 10556 heartbeater.cc:511] Master 127.0.98.62:35653 requested a full tablet report, sending...
I20260926 09:33:22.204916 10106 ts_manager.cc:194] Registered new tserver with Master: 85ed033469d745528e8c4de4f7417b88 (127.0.98.3:34849)
I20260926 09:33:22.209554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 10428
I20260926 09:33:22.209666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:33:22.209856 392 external_mini_cluster.cc:1428] Setting key 3e59a69d0fd1eb17579c1882b7af9b86
I20260926 09:33:22.210832 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 09:33:22.212635 10145 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:33:22.212196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.0.98.62:35653)
0926 09:33:22.212273 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:22.212275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:22.212369 (+ 94us) server_negotiation.cc:407] Connection header received
0926 09:33:22.212392 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:22.212393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:22.212397 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:22.212443 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:22.212570 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52050: BlockingRecv error: recv got EOF from 127.0.0.1:52050 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 09:33:22.212673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 10166
I20260926 09:33:22.217063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 10297
I20260926 09:33:22.221424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 10428
I20260926 09:33:22.225734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 10075
2026-09-26T09:33:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (542 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:33:22 dist-test-slave-21h8 krb5kdc[10566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:33:22 dist-test-slave-21h8 krb5kdc[10566](info): set up 2 sockets
Sep 26 09:33:22 dist-test-slave-21h8 krb5kdc[10566](info): commencing operation
krb5kdc: starting...
W20260926 09:33:23.292373 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:33:23 dist-test-slave-21h8 krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:33:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:33:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:33:23.411660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:23.491247 10579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:23.491570 10579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:23.491609 10579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:23.492751 10579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:33:23.492822 10579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:23.492846 10579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:23.492867 10579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:33:23.492897 10579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:33:23.494541 10579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:23.494827 10579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:23.495095 10579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:23.500162 10584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:23.500206 10587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:23.500356 10579 server_base.cc:1034] running on GCE node
W20260926 09:33:23.500415 10585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:23.500638 10579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:23.500974 10579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:23.502134 10579 hybrid_clock.cc:648] HybridClock initialized: now 1790415203502117 us; error 31 us; skew 500 ppm
Sep 26 09:33:23 dist-test-slave-21h8 krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:23.504761 10579 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:33:23.505316 10579 webserver.cc:460] Webserver started at http://127.0.98.62:38831/ using document root <none> and password file <none>
I20260926 09:33:23.505519 10579 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:23.505559 10579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:23.505645 10579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:23.506628 10579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "b7fa07e620654c0c8e6911211ec1fc16"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "5eea34476cb41b440e26631074ab512c"
server_key_iv: "23e174b975bab0e9d06329cb175cd156"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.506919 10579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7fa07e620654c0c8e6911211ec1fc16"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "5eea34476cb41b440e26631074ab512c"
server_key_iv: "23e174b975bab0e9d06329cb175cd156"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.508234 10579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:33:23.508906 10594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:23.509083 10579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:33:23.509188 10579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "b7fa07e620654c0c8e6911211ec1fc16"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "5eea34476cb41b440e26631074ab512c"
server_key_iv: "23e174b975bab0e9d06329cb175cd156"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.509261 10579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:23.522337 10579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:23.524155 10579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:23.524309 10579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:23.528193 10579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46565
I20260926 09:33:23.528223 10646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46565 every 8 connection(s)
I20260926 09:33:23.528558 10579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:33:23.529495 10647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:23.531718 10647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16: Bootstrap starting.
I20260926 09:33:23.532223 10647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:23.532454 10647 log.cc:824] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:23.533061 10647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16: No bootstrap required, opened a new log
I20260926 09:33:23.534546 10647 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa07e620654c0c8e6911211ec1fc16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46565 } }
I20260926 09:33:23.534677 10647 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:23.534698 10647 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fa07e620654c0c8e6911211ec1fc16, State: Initialized, Role: FOLLOWER
I20260926 09:33:23.534844 10647 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa07e620654c0c8e6911211ec1fc16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46565 } }
I20260926 09:33:23.534925 10647 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:33:23.534948 10647 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:33:23.534976 10647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:23.535715 10647 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa07e620654c0c8e6911211ec1fc16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46565 } }
I20260926 09:33:23.535800 10647 leader_election.cc:302] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7fa07e620654c0c8e6911211ec1fc16; no voters:
I20260926 09:33:23.535944 10647 leader_election.cc:288] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:33:23.535969 10650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:23.536079 10650 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [term 1 LEADER]: Becoming Leader. State: Replica: b7fa07e620654c0c8e6911211ec1fc16, State: Running, Role: LEADER
I20260926 09:33:23.536352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 10579
I20260926 09:33:23.536463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:33:23.536629 392 external_mini_cluster.cc:1428] Setting key 74c01e6d469e316e240c493a5e817b06
I20260926 09:33:23.536857 10650 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa07e620654c0c8e6911211ec1fc16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46565 } }
I20260926 09:33:23.537645 10652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7fa07e620654c0c8e6911211ec1fc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa07e620654c0c8e6911211ec1fc16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46565 } } }
I20260926 09:33:23.537725 10652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:23.536199 10647 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:33:23.537994 10651 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7fa07e620654c0c8e6911211ec1fc16. Latest consensus state: current_term: 1 leader_uuid: "b7fa07e620654c0c8e6911211ec1fc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fa07e620654c0c8e6911211ec1fc16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46565 } } }
I20260926 09:33:23.538118 10651 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16 [sys.catalog]: This master's current role is: LEADER
I20260926 09:33:23.538213 10656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:33:23.538816 10656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:33:23.540371 10656 catalog_manager.cc:1347] Generated new cluster ID: 3b0f6093b9c74c7a86976c97da8d8b6c
I20260926 09:33:23.540423 10656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:33:23 dist-test-slave-21h8 krb5kdc[10566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:33:23.549279 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:23.537076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.98.62:46565)
0926 09:33:23.537245 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:23.537257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:23.537390 (+ 133us) server_negotiation.cc:407] Connection header received
0926 09:33:23.537684 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:23.537692 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:23.537783 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:23.537957 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:23.538882 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.540398 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.541471 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.542079 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.542194 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:23.544974 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:23.544979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:23.544980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:23.545000 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:23.547665 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:23.548041 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:23.548044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:23.548045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:23.548167 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:23.548374 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:23.548375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:23.548375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:23.548683 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:23.548855 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:23.548935 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:33:23.549071 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260926 09:33:23.552063 10656 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:33:23.552750 10656 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:33:23.566910 10656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16: Generated new TSK 0
I20260926 09:33:23.567103 10656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:33:23.593971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46565
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:23.673674 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:23.673880 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:23.673903 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:23.675060 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:23.675105 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:23.675132 10673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:33:23.676635 10673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:23.676870 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:23.677109 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:23.682075 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:23.682075 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:23.682193 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:23.682155 10673 server_base.cc:1034] running on GCE node
I20260926 09:33:23.682487 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:23.682773 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:23.683938 10673 hybrid_clock.cc:648] HybridClock initialized: now 1790415203683931 us; error 36 us; skew 500 ppm
Sep 26 09:33:23 dist-test-slave-21h8 krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:23.686424 10673 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:33:23.686856 10673 webserver.cc:460] Webserver started at http://127.0.98.1:44543/ using document root <none> and password file <none>
I20260926 09:33:23.687099 10673 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:23.687170 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:23.687364 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:23.688400 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "e6c5516329a44a82b63695e30ba700f6"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "2c0f05599258e25e02fa867934f759ff"
server_key_iv: "9a5fcbf7437fe833e0075e23ada54998"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.688755 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e6c5516329a44a82b63695e30ba700f6"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "2c0f05599258e25e02fa867934f759ff"
server_key_iv: "9a5fcbf7437fe833e0075e23ada54998"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.690021 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:33:23.690727 10688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:23.690896 10673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:33:23.690994 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "e6c5516329a44a82b63695e30ba700f6"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "2c0f05599258e25e02fa867934f759ff"
server_key_iv: "9a5fcbf7437fe833e0075e23ada54998"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.691077 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:23.704066 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:23.705837 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:23.705992 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:23.706205 10673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:23.706521 10673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:23.706575 10673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:23.706614 10673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:23.706642 10673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:23.712428 10673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36357
I20260926 09:33:23.712499 10801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36357 every 8 connection(s)
I20260926 09:33:23.712809 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:33:23.718802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 10673
I20260926 09:33:23.718901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
Sep 26 09:33:23 dist-test-slave-21h8 krb5kdc[10566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:33:23.719090 392 external_mini_cluster.cc:1428] Setting key 06252f73b872c87428d0ac531edd73d5
I20260926 09:33:23.722316 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:23.713569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35681 (local address 127.0.98.62:46565)
0926 09:33:23.713616 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:23.713618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:23.713829 (+ 211us) server_negotiation.cc:407] Connection header received
0926 09:33:23.714358 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:23.714360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:23.714372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:23.714421 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:23.715352 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.715860 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.716832 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.717052 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.717137 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:23.719551 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:23.719553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:23.719553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:23.719560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:23.720929 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:23.721381 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:23.721382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:23.721382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:23.721449 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:23.721674 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:23.721675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:23.721675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:23.721807 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:23.721881 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:23.722129 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 09:33:23.722217 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:33:23.722455 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:23.713533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46565 (local address 127.0.98.1:35681)
0926 09:33:23.713707 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:23.713747 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:23.714243 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:23.714487 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:23.714491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:23.715048 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:23.715231 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:23.715236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.715957 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.715958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:23.716742 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:23.716744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.717147 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.717148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:23.717269 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:23.717809 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:23.717817 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:23.719453 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:23.721019 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:23.721026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:23.721029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:23.721305 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:23.721520 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:23.721520 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:23.721520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:23.721613 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:23.721871 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:23.721875 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:23.721982 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 09:33:23.722088 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:33:23.722236 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260926 09:33:23.722903 10802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46565
I20260926 09:33:23.723001 10802 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:23.723184 10802 heartbeater.cc:511] Master 127.0.98.62:46565 requested a full tablet report, sending...
I20260926 09:33:23.723773 10611 ts_manager.cc:194] Registered new tserver with Master: e6c5516329a44a82b63695e30ba700f6 (127.0.98.1:36357)
I20260926 09:33:23.724743 10611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35681
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:33:23.743602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46565
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:23.827181 10807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:23.827406 10807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:23.827430 10807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:23.828523 10807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:23.828557 10807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:23.828583 10807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:33:23.829943 10807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:23.830118 10807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:23.830286 10807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:23.835142 10812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:23.835162 10813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:23.835142 10815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:23.835479 10807 server_base.cc:1034] running on GCE node
I20260926 09:33:23.835687 10807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:23.835887 10807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:23.837030 10807 hybrid_clock.cc:648] HybridClock initialized: now 1790415203837016 us; error 29 us; skew 500 ppm
Sep 26 09:33:23 dist-test-slave-21h8 krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:23.839918 10807 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:33:23.840355 10807 webserver.cc:460] Webserver started at http://127.0.98.2:43739/ using document root <none> and password file <none>
I20260926 09:33:23.840597 10807 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:23.840641 10807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:23.840781 10807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:23.841820 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "6d5f9925c86743a68181fc22326b67b5"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "825997a5db95c19f11d66eed20de695c"
server_key_iv: "3ecb4ebc27c5158e3a61926521a516aa"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.842185 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6d5f9925c86743a68181fc22326b67b5"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "825997a5db95c19f11d66eed20de695c"
server_key_iv: "3ecb4ebc27c5158e3a61926521a516aa"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.843520 10807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:33:23.844244 10822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:23.844424 10807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:33:23.844528 10807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "6d5f9925c86743a68181fc22326b67b5"
format_stamp: "Formatted at 2026-09-26 09:33:23 on dist-test-slave-21h8"
server_key: "825997a5db95c19f11d66eed20de695c"
server_key_iv: "3ecb4ebc27c5158e3a61926521a516aa"
server_key_version: "encryptionkey@0"
I20260926 09:33:23.844605 10807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:23.880075 10807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:23.881943 10807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:23.882112 10807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:23.882377 10807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:23.882692 10807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:23.882745 10807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:23.882787 10807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:23.882812 10807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:23.888527 10807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44459
I20260926 09:33:23.888569 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44459 every 8 connection(s)
I20260926 09:33:23.888892 10807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:33:23 dist-test-slave-21h8 krb5kdc[10566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:33:23.898052 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:23.889638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53835 (local address 127.0.98.62:46565)
0926 09:33:23.889709 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:23.889711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:23.889893 (+ 182us) server_negotiation.cc:407] Connection header received
0926 09:33:23.890384 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:23.890385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:23.890395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:23.890450 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:23.891482 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.891971 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.892978 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.893150 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.893200 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:23.895454 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:23.895456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:23.895456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:23.895463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:23.896948 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:23.897289 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:23.897289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:23.897290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:23.897347 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:23.897550 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:23.897551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:23.897551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:23.897648 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:23.897696 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:23.897927 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 09:33:23.897987 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:33:23.898211 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:23.889604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46565 (local address 127.0.98.2:53835)
0926 09:33:23.889758 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:23.889809 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:23.890301 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:23.890515 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:23.890518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:23.891110 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:23.891381 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:23.891387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.892050 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.892050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:23.892904 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:23.892906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:23.893219 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:23.893220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:23.893349 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:23.893821 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:23.893829 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:23.895363 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:23.897017 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:23.897022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:23.897023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:23.897227 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:23.897404 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:23.897405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:23.897405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:23.897498 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:23.897701 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:23.897705 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:23.897821 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 09:33:23.897901 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:33:23.898014 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260926 09:33:23.898696 10936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46565
I20260926 09:33:23.898814 10936 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:23.899009 10936 heartbeater.cc:511] Master 127.0.98.62:46565 requested a full tablet report, sending...
I20260926 09:33:23.899056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 10807
I20260926 09:33:23.899144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:33:23.899384 392 external_mini_cluster.cc:1428] Setting key a873bd8ff1bfebb53bfc44c70af44376
I20260926 09:33:23.899461 10611 ts_manager.cc:194] Registered new tserver with Master: 6d5f9925c86743a68181fc22326b67b5 (127.0.98.2:44459)
I20260926 09:33:23.899976 10611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53835
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:33:23.923579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46565
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:33:24.002468 10941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:33:24.002667 10941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:33:24.002691 10941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:33:24.003886 10941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:33:24.003929 10941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:33:24.003957 10941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:33:24.005383 10941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:33:24.005632 10941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:33:24.005859 10941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:33:24.010790 10947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:24.010885 10946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:33:24.010812 10949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:33:24.010941 10941 server_base.cc:1034] running on GCE node
I20260926 09:33:24.011168 10941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:33:24.011523 10941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:33:24.012668 10941 hybrid_clock.cc:648] HybridClock initialized: now 1790415204012662 us; error 36 us; skew 500 ppm
Sep 26 09:33:24 dist-test-slave-21h8 krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:33:24.015002 10941 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:33:24.015553 10941 webserver.cc:460] Webserver started at http://127.0.98.3:35705/ using document root <none> and password file <none>
I20260926 09:33:24.015841 10941 fs_manager.cc:359] Metadata directory not provided
I20260926 09:33:24.015908 10941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:33:24.016038 10941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:33:24.017047 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f6d449bf7654a199b0042b00ad858ef"
format_stamp: "Formatted at 2026-09-26 09:33:24 on dist-test-slave-21h8"
server_key: "ca0ef9534b06d744a86a6ffd821c993b"
server_key_iv: "e67860ae2a81de544b0a9ff71c0efd4d"
server_key_version: "encryptionkey@0"
I20260926 09:33:24.017414 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f6d449bf7654a199b0042b00ad858ef"
format_stamp: "Formatted at 2026-09-26 09:33:24 on dist-test-slave-21h8"
server_key: "ca0ef9534b06d744a86a6ffd821c993b"
server_key_iv: "e67860ae2a81de544b0a9ff71c0efd4d"
server_key_version: "encryptionkey@0"
I20260926 09:33:24.018675 10941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:33:24.019418 10956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:24.019590 10941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:24.019685 10941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "6f6d449bf7654a199b0042b00ad858ef"
format_stamp: "Formatted at 2026-09-26 09:33:24 on dist-test-slave-21h8"
server_key: "ca0ef9534b06d744a86a6ffd821c993b"
server_key_iv: "e67860ae2a81de544b0a9ff71c0efd4d"
server_key_version: "encryptionkey@0"
I20260926 09:33:24.019765 10941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:33:24.034745 10941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:33:24.036655 10941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:33:24.036815 10941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:33:24.037050 10941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:33:24.037416 10941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:33:24.037467 10941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:24.037503 10941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:33:24.037671 10941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:33:24.043633 10941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38711
I20260926 09:33:24.043685 11069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38711 every 8 connection(s)
I20260926 09:33:24.044013 10941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:33:24.047783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 10941
I20260926 09:33:24.047860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:33:24.048046 392 external_mini_cluster.cc:1428] Setting key e024d379612cfd6e824045d7a836b311
Sep 26 09:33:24 dist-test-slave-21h8 krb5kdc[10566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:33:24.053447 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.044743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53993 (local address 127.0.98.62:46565)
0926 09:33:24.044809 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:24.044811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:24.045028 (+ 217us) server_negotiation.cc:407] Connection header received
0926 09:33:24.045549 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:24.045550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:24.045562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:24.045620 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:24.046600 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.047094 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.048488 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.048803 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.048872 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.050987 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:24.050988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:24.050989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:24.050995 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:24.052262 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:24.052629 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:24.052630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:24.052630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:24.052678 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:24.052883 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:24.052883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:24.052884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:24.052979 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:24.053029 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:24.053319 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 09:33:24.053383 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:33:24.053681 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.044736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46565 (local address 127.0.98.3:53993)
0926 09:33:24.044908 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:24.044943 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:24.045445 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:24.045715 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:24.045718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:24.046284 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:33:24.046475 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.046479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.047271 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.047271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.048393 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.048396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.048888 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.048889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.049029 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.049570 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:33:24.049583 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:33:24.050907 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:33:24.052345 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:24.052350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:24.052352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:24.052563 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:24.052735 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:33:24.052736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:33:24.052736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:33:24.052828 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:33:24.053058 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:33:24.053064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:33:24.053187 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 09:33:24.053293 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:33:24.053413 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260926 09:33:24.054198 11070 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46565
I20260926 09:33:24.054279 11070 heartbeater.cc:464] Registering TS with master...
I20260926 09:33:24.054462 11070 heartbeater.cc:511] Master 127.0.98.62:46565 requested a full tablet report, sending...
I20260926 09:33:24.054872 10611 ts_manager.cc:194] Registered new tserver with Master: 6f6d449bf7654a199b0042b00ad858ef (127.0.98.3:38711)
I20260926 09:33:24.055238 10611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53993
I20260926 09:33:24.059809 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:33:24.066951 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.061056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.98.62:46565)
0926 09:33:24.061147 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:24.061149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:24.061186 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:33:24.061204 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:24.061205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:24.061213 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:24.061251 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:33:24.061750 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.062204 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.062870 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.063037 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.063085 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.064165 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:33:24.064166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:33:24.064166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:33:24.064172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:33:24.065908 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:24.066205 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:24.066206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:24.066206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:24.066255 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:33:24.066416 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:33:24.066417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:33:24.066417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:33:24.066543 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:33:24.066616 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:24.066758 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 09:33:24.066841 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 09:33:24.068360 10611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:33:24.069823 10611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:33:24.076434 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.072483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.0.98.2:44459)
0926 09:33:24.072664 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:24.072668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:24.072676 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:33:24.072705 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:24.072706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:24.072717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:24.073027 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:24.073119 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.074248 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.075530 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.075993 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.076092 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.076150 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:24.076310 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 09:33:24.076348 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260926 09:33:24.076436 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.072254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44459 (local address 127.0.0.1:58958)
0926 09:33:24.072467 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:24.072477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:24.072516 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:24.072879 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:24.072880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:24.072881 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:24.072923 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.072927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.074345 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.074346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.075452 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.075454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.076095 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.076097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.076173 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.076186 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 09:33:24.076293 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 09:33:24.076346 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260926 09:33:24.076488 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.072146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.0.98.3:38711)
0926 09:33:24.072310 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:24.072315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:24.072650 (+ 335us) server_negotiation.cc:407] Connection header received
0926 09:33:24.072686 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:24.072688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:24.072701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:24.072762 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:24.072964 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.074102 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.075267 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.076008 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.076085 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.076144 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:24.076362 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 09:33:24.076404 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260926 09:33:24.076442 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.072067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36357 (local address 127.0.0.1:55810)
0926 09:33:24.072326 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:24.072370 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:24.072448 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:24.073069 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:24.073070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:24.073072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:24.073113 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.073115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.074021 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.074023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.075230 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.075233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.076181 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.076182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.076251 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.076257 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:33:24.076306 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:33:24.076355 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260926 09:33:24.076735 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.072071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38711 (local address 127.0.0.1:44670)
0926 09:33:24.072358 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:24.072376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:24.072450 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:24.072789 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:24.072792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:24.072795 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:24.072869 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.072872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.074221 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.074222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.075169 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.075171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.076181 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.076182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.076251 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.076255 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:33:24.076564 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 09:33:24.076614 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260926 09:33:24.077237 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.072159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.0.98.1:36357)
0926 09:33:24.072388 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:24.072393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:24.072859 (+ 466us) server_negotiation.cc:407] Connection header received
0926 09:33:24.072904 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:24.072906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:24.072921 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:24.072990 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:24.073201 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.073917 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.075366 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.076053 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.076109 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.076154 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:24.076479 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 09:33:24.077151 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260926 09:33:24.077172 10870 tablet_service.cc:1467] Processing CreateTablet for tablet 425959e71fe54caa961e5a32115c1ba5 (DEFAULT_TABLE table=test-table [id=d6165c601eba4aeab830c4c25a715df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:24.077515 10870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425959e71fe54caa961e5a32115c1ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:24.077823 10736 tablet_service.cc:1467] Processing CreateTablet for tablet 425959e71fe54caa961e5a32115c1ba5 (DEFAULT_TABLE table=test-table [id=d6165c601eba4aeab830c4c25a715df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:24.078118 10736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425959e71fe54caa961e5a32115c1ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:24.080010 11004 tablet_service.cc:1467] Processing CreateTablet for tablet 425959e71fe54caa961e5a32115c1ba5 (DEFAULT_TABLE table=test-table [id=d6165c601eba4aeab830c4c25a715df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:33:24.080264 11087 tablet_bootstrap.cc:492] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Bootstrap starting.
I20260926 09:33:24.080322 11004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425959e71fe54caa961e5a32115c1ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:33:24.080829 11087 tablet_bootstrap.cc:654] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:24.081146 11088 tablet_bootstrap.cc:492] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6: Bootstrap starting.
I20260926 09:33:24.081466 11087 log.cc:824] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:24.081842 11088 tablet_bootstrap.cc:654] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:24.082410 11087 tablet_bootstrap.cc:492] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: No bootstrap required, opened a new log
I20260926 09:33:24.082396 11088 log.cc:824] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:24.082480 11087 ts_tablet_manager.cc:1397] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:33:24.083201 11088 tablet_bootstrap.cc:492] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6: No bootstrap required, opened a new log
I20260926 09:33:24.083272 11088 ts_tablet_manager.cc:1397] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:33:24.083442 11091 tablet_bootstrap.cc:492] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: Bootstrap starting.
I20260926 09:33:24.084065 11091 tablet_bootstrap.cc:654] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: Neither blocks nor log segments found. Creating new log.
I20260926 09:33:24.084474 11087 raft_consensus.cc:348] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.084604 11091 log.cc:824] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: Log is configured to *not* fsync() on all Append() calls
I20260926 09:33:24.084652 11087 raft_consensus.cc:374] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:24.084689 11087 raft_consensus.cc:729] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5f9925c86743a68181fc22326b67b5, State: Initialized, Role: FOLLOWER
I20260926 09:33:24.084818 11087 consensus_queue.cc:260] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.085103 11087 ts_tablet_manager.cc:1428] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:33:24.085150 10936 heartbeater.cc:503] Master 127.0.98.62:46565 was elected leader, sending a full tablet report...
I20260926 09:33:24.085222 11088 raft_consensus.cc:348] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.085412 11088 raft_consensus.cc:374] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:24.085459 11088 raft_consensus.cc:729] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c5516329a44a82b63695e30ba700f6, State: Initialized, Role: FOLLOWER
I20260926 09:33:24.085593 11088 consensus_queue.cc:260] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.085734 11091 tablet_bootstrap.cc:492] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: No bootstrap required, opened a new log
I20260926 09:33:24.085793 11091 ts_tablet_manager.cc:1397] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:33:24.085860 11088 ts_tablet_manager.cc:1428] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:33:24.085889 10802 heartbeater.cc:503] Master 127.0.98.62:46565 was elected leader, sending a full tablet report...
I20260926 09:33:24.087591 11091 raft_consensus.cc:348] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.087757 11091 raft_consensus.cc:374] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:33:24.087805 11091 raft_consensus.cc:729] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6d449bf7654a199b0042b00ad858ef, State: Initialized, Role: FOLLOWER
I20260926 09:33:24.087929 11091 consensus_queue.cc:260] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.088161 11091 ts_tablet_manager.cc:1428] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:33:24.088325 11070 heartbeater.cc:503] Master 127.0.98.62:46565 was elected leader, sending a full tablet report...
W20260926 09:33:24.139465 10937 tablet.cc:2369] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:33:24.213626 10803 tablet.cc:2369] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:33:24.214418 11095 raft_consensus.cc:482] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:33:24.214553 11095 raft_consensus.cc:504] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.214963 11095 leader_election.cc:288] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6c5516329a44a82b63695e30ba700f6 (127.0.98.1:36357), 6d5f9925c86743a68181fc22326b67b5 (127.0.98.2:44459)
I20260926 09:33:24.218008 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.215183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36357 (local address 127.0.98.3:33157)
0926 09:33:24.215224 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:24.215237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:24.215305 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:24.215497 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:24.215498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:24.215499 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:24.215574 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.215576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.216241 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.216242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.217188 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.217191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.217709 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.217710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.217801 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.217804 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:33:24.217894 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:33:24.217937 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 09:33:24.218007 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.215265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33157 (local address 127.0.98.1:36357)
0926 09:33:24.215357 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:24.215360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:24.215369 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:33:24.215405 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:24.215407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:24.215419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:24.215493 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:24.215644 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.216162 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.217254 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.217627 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.217712 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.217722 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:24.217906 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 09:33:24.217944 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 09:33:24.218258 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.215227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38337 (local address 127.0.98.2:44459)
0926 09:33:24.215287 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:24.215290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:24.215572 (+ 282us) server_negotiation.cc:407] Connection header received
0926 09:33:24.215650 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:24.215651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:24.215664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:24.215713 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:33:24.215890 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.216349 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.217481 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.217864 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.217950 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.217959 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:24.218141 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 09:33:24.218195 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:33:24.218382 10756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425959e71fe54caa961e5a32115c1ba5" candidate_uuid: "6f6d449bf7654a199b0042b00ad858ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c5516329a44a82b63695e30ba700f6" is_pre_election: true
I20260926 09:33:24.218465 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.215185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44459 (local address 127.0.98.3:38337)
0926 09:33:24.215498 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:24.215506 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:33:24.215584 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:33:24.215726 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:33:24.215727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:33:24.215728 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:33:24.215801 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.215803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.216429 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.216430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.217395 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:33:24.217397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.217953 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.217954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:33:24.218024 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.218033 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 09:33:24.218338 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 09:33:24.218385 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":118,"thread_start_us":113,"threads_started":1}
I20260926 09:33:24.218547 10756 raft_consensus.cc:2454] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6d449bf7654a199b0042b00ad858ef in term 0.
I20260926 09:33:24.218786 10957 leader_election.cc:302] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6d449bf7654a199b0042b00ad858ef, e6c5516329a44a82b63695e30ba700f6; no voters:
I20260926 09:33:24.218779 10890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425959e71fe54caa961e5a32115c1ba5" candidate_uuid: "6f6d449bf7654a199b0042b00ad858ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5f9925c86743a68181fc22326b67b5" is_pre_election: true
I20260926 09:33:24.218921 10890 raft_consensus.cc:2454] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6d449bf7654a199b0042b00ad858ef in term 0.
I20260926 09:33:24.218911 11095 raft_consensus.cc:2781] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:33:24.218966 11095 raft_consensus.cc:482] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:33:24.218991 11095 raft_consensus.cc:3037] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:24.219677 11095 raft_consensus.cc:504] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.219816 11095 leader_election.cc:288] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [CANDIDATE]: Term 1 election: Requested vote from peers e6c5516329a44a82b63695e30ba700f6 (127.0.98.1:36357), 6d5f9925c86743a68181fc22326b67b5 (127.0.98.2:44459)
I20260926 09:33:24.219967 10890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425959e71fe54caa961e5a32115c1ba5" candidate_uuid: "6f6d449bf7654a199b0042b00ad858ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5f9925c86743a68181fc22326b67b5"
I20260926 09:33:24.219987 10756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425959e71fe54caa961e5a32115c1ba5" candidate_uuid: "6f6d449bf7654a199b0042b00ad858ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c5516329a44a82b63695e30ba700f6"
I20260926 09:33:24.220043 10890 raft_consensus.cc:3037] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:24.220057 10756 raft_consensus.cc:3037] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:33:24.220722 10756 raft_consensus.cc:2454] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6d449bf7654a199b0042b00ad858ef in term 1.
I20260926 09:33:24.220865 10890 raft_consensus.cc:2454] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6d449bf7654a199b0042b00ad858ef in term 1.
I20260926 09:33:24.220943 10957 leader_election.cc:302] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6d449bf7654a199b0042b00ad858ef, e6c5516329a44a82b63695e30ba700f6; no voters:
I20260926 09:33:24.221043 11095 raft_consensus.cc:2781] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:33:24.221204 11095 raft_consensus.cc:686] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 1 LEADER]: Becoming Leader. State: Replica: 6f6d449bf7654a199b0042b00ad858ef, State: Running, Role: LEADER
I20260926 09:33:24.221310 11095 consensus_queue.cc:237] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } }
I20260926 09:33:24.222158 10610 catalog_manager.cc:5495] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f6d449bf7654a199b0042b00ad858ef (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6f6d449bf7654a199b0042b00ad858ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f6d449bf7654a199b0042b00ad858ef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 } health_report { overall_health: UNKNOWN } } }
I20260926 09:33:24.257845 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.255362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.0.98.3:38711)
0926 09:33:24.255411 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:33:24.255413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:33:24.255422 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:33:24.255472 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:33:24.255473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:33:24.255494 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:33:24.255586 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:33:24.255734 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.256211 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.257002 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:33:24.257202 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:33:24.257283 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:33:24.257468 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:33:24.257574 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:33:24.257667 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:33:24.257694 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 09:33:24.257717 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:33:24.259824 10756 raft_consensus.cc:1261] T 425959e71fe54caa961e5a32115c1ba5 P e6c5516329a44a82b63695e30ba700f6 [term 1 FOLLOWER]: Refusing update from remote peer 6f6d449bf7654a199b0042b00ad858ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:24.259843 10890 raft_consensus.cc:1261] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 1 FOLLOWER]: Refusing update from remote peer 6f6d449bf7654a199b0042b00ad858ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:33:24.260237 11095 consensus_queue.cc:1035] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c5516329a44a82b63695e30ba700f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:24.260422 11097 consensus_queue.cc:1035] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5f9925c86743a68181fc22326b67b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:33:24.263125 11102 mvcc.cc:205] Tried to move back new op lower bound from 7333540676645183488 to 7333540676491161600. Current Snapshot: MvccSnapshot[applied={T|T < 7333540676645183488}]
I20260926 09:33:24.263998 11101 mvcc.cc:205] Tried to move back new op lower bound from 7333540676645183488 to 7333540676491161600. Current Snapshot: MvccSnapshot[applied={T|T < 7333540676645183488}]
I20260926 09:33:24.266464 10611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:24.266577 10611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:33:24.267382 10611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7fa07e620654c0c8e6911211ec1fc16: Sending DeleteTablet for 3 replicas of tablet 425959e71fe54caa961e5a32115c1ba5
I20260926 09:33:24.267788 10736 tablet_service.cc:1514] Processing DeleteTablet for tablet 425959e71fe54caa961e5a32115c1ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55810
I20260926 09:33:24.267812 11004 tablet_service.cc:1514] Processing DeleteTablet for tablet 425959e71fe54caa961e5a32115c1ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44670
I20260926 09:33:24.268054 10870 tablet_service.cc:1514] Processing DeleteTablet for tablet 425959e71fe54caa961e5a32115c1ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:33:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58958
I20260926 09:33:24.268112 11110 tablet_replica.cc:330] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: stopping tablet replica
I20260926 09:33:24.268237 11110 raft_consensus.cc:2229] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:33:24.268400 11112 tablet_replica.cc:330] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: stopping tablet replica
I20260926 09:33:24.268457 11110 raft_consensus.cc:2258] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:24.268522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 10673
I20260926 09:33:24.268684 11112 raft_consensus.cc:2229] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:33:24.268867 11112 raft_consensus.cc:2258] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:33:24.269755 11112 ts_tablet_manager.cc:1905] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:24.269825 11110 ts_tablet_manager.cc:1905] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:33:24.271910 11112 ts_tablet_manager.cc:1918] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:33:24.272047 11112 log.cc:1196] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/425959e71fe54caa961e5a32115c1ba5
I20260926 09:33:24.272068 11110 ts_tablet_manager.cc:1918] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:33:24.272142 11110 log.cc:1196] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/425959e71fe54caa961e5a32115c1ba5
I20260926 09:33:24.272436 11110 ts_tablet_manager.cc:1939] T 425959e71fe54caa961e5a32115c1ba5 P 6f6d449bf7654a199b0042b00ad858ef: Deleting consensus metadata
I20260926 09:33:24.272548 11112 ts_tablet_manager.cc:1939] T 425959e71fe54caa961e5a32115c1ba5 P 6d5f9925c86743a68181fc22326b67b5: Deleting consensus metadata
I20260926 09:33:24.272881 10596 catalog_manager.cc:4841] TS 6f6d449bf7654a199b0042b00ad858ef (127.0.98.3:38711): tablet 425959e71fe54caa961e5a32115c1ba5 (table test-table [id=d6165c601eba4aeab830c4c25a715df2]) successfully deleted
I20260926 09:33:24.272995 10596 catalog_manager.cc:4841] TS 6d5f9925c86743a68181fc22326b67b5 (127.0.98.2:44459): tablet 425959e71fe54caa961e5a32115c1ba5 (table test-table [id=d6165c601eba4aeab830c4c25a715df2]) successfully deleted
W20260926 09:33:24.275978 10595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36357 (error 108)
I20260926 09:33:24.276170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 10807
I20260926 09:33:24.282282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 10941
I20260926 09:33:24.282307 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:33:24.282166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36357 (local address 127.0.0.1:55816)
0926 09:33:24.282229 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 09:33:24.282274 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 09:33:24.283094 10595 catalog_manager.cc:4584] TS e6c5516329a44a82b63695e30ba700f6 (127.0.98.1:36357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 425959e71fe54caa961e5a32115c1ba5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36357: connect: Connection refused (error 111)
I20260926 09:33:24.287096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 10579
2026-09-26T09:33:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2062 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:33:24 dist-test-slave-21h8 krb5kdc[11116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:33:24 dist-test-slave-21h8 krb5kdc[11116](info): set up 2 sockets
Sep 26 09:33:24 dist-test-slave-21h8 krb5kdc[11116](info): commencing operation
krb5kdc: starting...
W20260926 09:33:25.312289 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:33:25 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:33:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 09:33:25.477205 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5TYWI0/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/postgres -l logfile start
2026-09-26 09:33:27.838 UTC [11146] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 09:33:27.838 UTC [11146] LOG: listening on IPv4 address "127.0.98.20", port 46069
2026-09-26 09:33:27.842 UTC [11146] LOG: listening on Unix socket "/tmp/.s.PGSQL.46069"
2026-09-26 09:33:27.872 UTC [11149] LOG: database system was shut down at 2026-09-26 09:33:26 UTC
2026-09-26 09:33:27.876 UTC [11146] LOG: database system is ready to accept connections
I20260926 09:33:28.833077 392 mini_postgres.cc:96] Postgres bound to 46069
2026-09-26 09:33:28.839 UTC [11157] FATAL: database "slave" does not exist
127.0.98.20:46069 - accepting connections
I20260926 09:33:28.840215 392 mini_ranger.cc:161] Starting Ranger...
I20260926 09:33:28.855121 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 09:33:29.320356 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 09:33:29.320444 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-admin
2026-09-26 09:33:29,613 [I] DB FLAVOR :POSTGRES
2026-09-26 09:33:29,614 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:33:29,614 [I] Checking connection..
2026-09-26 09:33:29,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 09:33:29,869 [I] Checking connection passed.
2026-09-26 09:33:29,869 [I] --------- Verifying version history table ---------
2026-09-26 09:33:29,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:33:30,121 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:33:30,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:33:30,362 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:33:30,363 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 09:33:30,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 09:33:30.620 UTC [11221] WARNING: there is no transaction in progress
2026-09-26 09:33:30,626 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 09:33:30,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:33:30,860 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 09:33:30,860 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:33:30,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 09:33:31,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 09:33:31,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-21h8.c.gcp-upstream.internal','N') ;"
2026-09-26 09:33:31,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:33:31,863 [I] Table x_portal_user does not exist in database ranger
2026-09-26 09:33:31,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:33:32,112 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 09:33:32,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:33:32,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:33:32,572 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 09:33:32,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 09:33:33.523 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.540 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.556 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.571 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.588 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.605 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.659 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.664 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.677 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.687 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.698 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.711 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.721 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.726 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.732 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:33.738 UTC [11329] WARNING: there is no transaction in progress
2026-09-26 09:33:34,328 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 09:33:34,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 09:33:34,575 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 09:33:34,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:33:34,839 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 09:33:34,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:33:35,102 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 09:33:35,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:33:35,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:33:35,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-21h8.c.gcp-upstream.internal';"
2026-09-26 09:33:35,851 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 09:33:35,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 09:33:36,109 [I] DB_PATCHES have already been applied
I20260926 09:33:36.114868 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5TYWI0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 09:33:36.114950 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 09:33:36.117940 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42927
Sep 26, 2026 9:33:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:33:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:33:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:33:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:33:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:33:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:33:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 09:33:37.245316 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.405s user 0.000s sys 0.009s
Sep 26, 2026 9:33:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:33:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 9:33:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:33:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:33:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 09:33:59.041765 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 09:33:59 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415239, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 09:33:59.140645 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 09:33:59.151044 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 09:33:59.955180 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 09:33:59.955263 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms
I20260926 09:33:59.955286 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:46069
2026-09-26 09:34:00,126 [I] DB FLAVOR :POSTGRES
2026-09-26 09:34:00,126 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:34:00,126 [I] Checking connection
2026-09-26 09:34:00,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 09:34:00,384 [I] connection success
2026-09-26 09:34:00,385 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 09:34:00,385 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 09:34:00,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 09:34:00,649 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 09:34:00,649 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:34:00,649 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 09:34:00,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 09:34:00,930 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 09:34:01.399886 392 mini_ranger_kms.cc:310] Created kms service
I20260926 09:34:01.583969 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 09:34:01.629347 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 09:34:01 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415239, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:02.021027 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 09:34:02.021143 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 09:34:02.021167 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 09:34:02.024361 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11509
I20260926 09:34:02.024488 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54589
Sep 26, 2026 9:34:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:34:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:34:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:34:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:34:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:34:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 09:34:04.303440 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.163s user 0.013s sys 0.141s
I20260926 09:34:04.303548 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 09:34:04.303625 392 mini_ranger_kms.cc:401] 127.0.98.20:54589/kms/v1/keys
Sep 26 09:34:04 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790415244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 09:34:04 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 09:34:05 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 09:34:07 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:34:07.514802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42559
--ranger_config_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54589/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:34:07.598402 11560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:07.598621 11560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:07.598649 11560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:07.599920 11560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:34:07.599967 11560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:07.599983 11560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:07.599998 11560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:34:07.600013 11560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:34:07.601616 11560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54589/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42559
--ranger_config_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:07.601851 11560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:07.602118 11560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:07.608299 11565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:07.608420 11560 server_base.cc:1034] running on GCE node
W20260926 09:34:07.608301 11566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:07.608325 11568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:07.608810 11560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:07.609146 11560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:07.610322 11560 hybrid_clock.cc:648] HybridClock initialized: now 1790415247610309 us; error 29 us; skew 500 ppm
Sep 26 09:34:07 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:07.620062 11560 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:34:07.620685 11560 webserver.cc:460] Webserver started at http://127.0.98.62:33239/ using document root <none> and password file <none>
I20260926 09:34:07.620970 11560 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:07.621054 11560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:07.621222 11560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:34:07 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:07.670874 11560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "e8a78f5d1320468a93156f2d6268104d"
format_stamp: "Formatted at 2026-09-26 09:34:07 on dist-test-slave-21h8"
server_key: "6452be2267485a79f7b2ec4b311d2bc9"
server_key_iv: "56c5110e57a0197717f3f36f342199e6"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:07.671303 11560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8a78f5d1320468a93156f2d6268104d"
format_stamp: "Formatted at 2026-09-26 09:34:07 on dist-test-slave-21h8"
server_key: "6452be2267485a79f7b2ec4b311d2bc9"
server_key_iv: "56c5110e57a0197717f3f36f342199e6"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:07.672951 11560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:34:07.685122 11575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:07.685376 11560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:34:07.685484 11560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "e8a78f5d1320468a93156f2d6268104d"
format_stamp: "Formatted at 2026-09-26 09:34:07 on dist-test-slave-21h8"
server_key: "6452be2267485a79f7b2ec4b311d2bc9"
server_key_iv: "56c5110e57a0197717f3f36f342199e6"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:07.685628 11560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:07.705677 11560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:07.707768 11560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:07.707962 11560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:07.713177 11560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42559
I20260926 09:34:07.713203 11627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42559 every 8 connection(s)
I20260926 09:34:07.713541 11560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:34:07.714286 11628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:07.717311 11628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d: Bootstrap starting.
I20260926 09:34:07.718044 11628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:07.718391 11628 log.cc:824] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:07.719298 11628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d: No bootstrap required, opened a new log
I20260926 09:34:07.721211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 11560
I20260926 09:34:07.721314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:34:07.721477 392 external_mini_cluster.cc:1428] Setting key 4e7894084d627053dd98c6611b3701e3
I20260926 09:34:07.722013 11628 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a78f5d1320468a93156f2d6268104d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42559 } }
I20260926 09:34:07.722198 11628 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:07.722234 11628 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a78f5d1320468a93156f2d6268104d, State: Initialized, Role: FOLLOWER
I20260926 09:34:07.722388 11628 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a78f5d1320468a93156f2d6268104d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42559 } }
I20260926 09:34:07.722465 11628 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:34:07.722494 11628 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:34:07.722559 11628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:07.723176 11628 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a78f5d1320468a93156f2d6268104d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42559 } }
I20260926 09:34:07.723280 11628 leader_election.cc:302] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8a78f5d1320468a93156f2d6268104d; no voters:
I20260926 09:34:07.723491 11628 leader_election.cc:288] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:34:07.723542 11633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:34:07.723757 11633 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [term 1 LEADER]: Becoming Leader. State: Replica: e8a78f5d1320468a93156f2d6268104d, State: Running, Role: LEADER
I20260926 09:34:07.723819 11628 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:34:07.723910 11633 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a78f5d1320468a93156f2d6268104d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42559 } }
I20260926 09:34:07.724298 11633 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8a78f5d1320468a93156f2d6268104d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a78f5d1320468a93156f2d6268104d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42559 } } }
I20260926 09:34:07.724417 11633 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:07.724576 11633 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8a78f5d1320468a93156f2d6268104d. Latest consensus state: current_term: 1 leader_uuid: "e8a78f5d1320468a93156f2d6268104d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a78f5d1320468a93156f2d6268104d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42559 } } }
I20260926 09:34:07.724671 11633 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:07.724824 11642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:34:07.725342 11642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:34:07.728051 11642 catalog_manager.cc:1347] Generated new cluster ID: 860bb52a22b547fa900b87ec939aef8f
I20260926 09:34:07.728161 11642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:34:07 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:07.733409 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:07.721944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.0.98.62:42559)
0926 09:34:07.722116 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:07.722126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:07.722172 (+ 46us) server_negotiation.cc:407] Connection header received
0926 09:34:07.722333 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:07.722340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:07.722400 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:07.722553 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:07.723224 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:07.724629 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:07.725715 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:07.726772 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:07.729706 (+ 2934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:07.729818 (+ 112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:07.729820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:07.729821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:07.729829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:07.731635 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:07.732002 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:07.732005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:07.732005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:07.732087 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:07.732262 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:07.732262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:07.732263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:07.732542 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:07.733054 (+ 512us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:07.733117 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 09:34:07.733218 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260926 09:34:07.735630 11628 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 09:34:07.758049 11642 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:34:07.758975 11642 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:34:07.762872 11642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d: Generated new TSK 0
I20260926 09:34:07.763130 11642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:34:08.422583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42559
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:34:08.531111 11673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:08.531307 11673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:08.531427 11673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:08.532637 11673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:08.532701 11673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:08.532747 11673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:34:08.534240 11673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:08.534512 11673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:08.534752 11673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:08.539786 11679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:08.539825 11678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:08.539858 11681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:08.540207 11673 server_base.cc:1034] running on GCE node
I20260926 09:34:08.540412 11673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:08.540711 11673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:08.541900 11673 hybrid_clock.cc:648] HybridClock initialized: now 1790415248541884 us; error 27 us; skew 500 ppm
Sep 26 09:34:08 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:08.544957 11673 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:34:08.545416 11673 webserver.cc:460] Webserver started at http://127.0.98.1:45501/ using document root <none> and password file <none>
I20260926 09:34:08.545642 11673 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:08.545708 11673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:08.545857 11673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:34:08 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:08.561887 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "5cd6b236fd644141be5a8f52e28da355"
format_stamp: "Formatted at 2026-09-26 09:34:08 on dist-test-slave-21h8"
server_key: "cbb1f2d235446a73be494d7b77b6ea50"
server_key_iv: "b6ced32572424b75a68c9d50c7dcad11"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:08.562309 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5cd6b236fd644141be5a8f52e28da355"
format_stamp: "Formatted at 2026-09-26 09:34:08 on dist-test-slave-21h8"
server_key: "cbb1f2d235446a73be494d7b77b6ea50"
server_key_iv: "b6ced32572424b75a68c9d50c7dcad11"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:08.564510 11673 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 09:34:08.586615 11690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:08.586894 11673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 09:34:08.587062 11673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "5cd6b236fd644141be5a8f52e28da355"
format_stamp: "Formatted at 2026-09-26 09:34:08 on dist-test-slave-21h8"
server_key: "cbb1f2d235446a73be494d7b77b6ea50"
server_key_iv: "b6ced32572424b75a68c9d50c7dcad11"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:08.587258 11673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:08.607822 11673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:08.610373 11673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:08.610594 11673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:08.610908 11673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:08.611364 11673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:08.611454 11673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:08.611549 11673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:08.611625 11673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:08.618958 11673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44813
I20260926 09:34:08.619690 11673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:34:08.620626 11803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44813 every 8 connection(s)
Sep 26 09:34:08 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:08.629318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 11673
I20260926 09:34:08.629420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:34:08.629643 392 external_mini_cluster.cc:1428] Setting key e19bd8f81f6e4059946367515d9cc07a
I20260926 09:34:08.632639 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:08.621015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42559 (local address 127.0.98.1:46707)
0926 09:34:08.621207 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:08.621255 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:08.621890 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:08.622249 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:08.622253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:08.623150 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:08.623422 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:08.623427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:08.624682 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:08.624683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:08.625988 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:08.625990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:08.626390 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:08.626391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:08.626529 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:08.627078 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:08.627092 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:08.628611 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:08.630658 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:08.630665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:08.630667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:08.630884 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:08.631106 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:08.631107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:08.631108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:08.631210 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:08.631531 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:08.631538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:08.631677 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 09:34:08.631792 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 09:34:08.632368 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260926 09:34:08.633232 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:08.621049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46707 (local address 127.0.98.62:42559)
0926 09:34:08.621296 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:08.621299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:08.621339 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:34:08.622045 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:08.622046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:08.622061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:08.622150 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:08.623545 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:08.624477 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:08.626068 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:08.626294 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:08.626382 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:08.628709 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:08.628710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:08.628711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:08.628719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:08.630569 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:08.630963 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:08.630965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:08.630965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:08.631026 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:08.631293 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:08.631294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:08.631295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:08.631453 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:08.631529 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:08.633025 (+ 1496us) server_negotiation.cc:299] Negotiation successful
0926 09:34:08.633116 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260926 09:34:08.633963 11804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42559
I20260926 09:34:08.634100 11804 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:08.634339 11804 heartbeater.cc:511] Master 127.0.98.62:42559 requested a full tablet report, sending...
I20260926 09:34:08.634938 11592 ts_manager.cc:194] Registered new tserver with Master: 5cd6b236fd644141be5a8f52e28da355 (127.0.98.1:44813)
I20260926 09:34:08.635910 11592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46707
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:34:08.658684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42559
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:34:08 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790415248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 09:34:08.782565 11810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:08.782941 11810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:08.783007 11810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:08.784759 11810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:08.784821 11810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:08.784858 11810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:34:08.787041 11810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:08.787281 11810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:08.787609 11810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:08.793789 11817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:08.793921 11818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:08.794027 11820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:08.794678 11810 server_base.cc:1034] running on GCE node
I20260926 09:34:08.794888 11810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:08.795138 11810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:08.796316 11810 hybrid_clock.cc:648] HybridClock initialized: now 1790415248796288 us; error 40 us; skew 500 ppm
Sep 26 09:34:08 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:08.799428 11810 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:34:08.799912 11810 webserver.cc:460] Webserver started at http://127.0.98.2:38869/ using document root <none> and password file <none>
I20260926 09:34:08.800205 11810 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:08.800276 11810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:08.800464 11810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:34:08 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:08.817078 11810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "a23feade64334fcdb5e6ce128abf0d0c"
format_stamp: "Formatted at 2026-09-26 09:34:08 on dist-test-slave-21h8"
server_key: "639591b6df32bb6a0f0e1ce59e35dd4f"
server_key_iv: "19ce6249510c8356f9939aec01d9b82b"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:08.817561 11810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a23feade64334fcdb5e6ce128abf0d0c"
format_stamp: "Formatted at 2026-09-26 09:34:08 on dist-test-slave-21h8"
server_key: "639591b6df32bb6a0f0e1ce59e35dd4f"
server_key_iv: "19ce6249510c8356f9939aec01d9b82b"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:08.819017 11810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:34:08.838917 11828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:08.839213 11810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 09:34:08.839334 11810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "a23feade64334fcdb5e6ce128abf0d0c"
format_stamp: "Formatted at 2026-09-26 09:34:08 on dist-test-slave-21h8"
server_key: "639591b6df32bb6a0f0e1ce59e35dd4f"
server_key_iv: "19ce6249510c8356f9939aec01d9b82b"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:08.839442 11810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:08.856638 11810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:08.858841 11810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:08.859004 11810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:08.859242 11810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:08.859661 11810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:08.859711 11810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:08.859745 11810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:08.859771 11810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:08.867519 11810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40067
I20260926 09:34:08.867610 11941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40067 every 8 connection(s)
I20260926 09:34:08.867938 11810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:34:08 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:08.877939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 11810
I20260926 09:34:08.878044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:34:08.878221 392 external_mini_cluster.cc:1428] Setting key 49bfbb9cf5189140252436cfb41ff765
I20260926 09:34:08.878265 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:08.868675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58941 (local address 127.0.98.62:42559)
0926 09:34:08.868760 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:08.868762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:08.869022 (+ 260us) server_negotiation.cc:407] Connection header received
0926 09:34:08.869531 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:08.869532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:08.869544 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:08.869625 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:08.870142 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:08.870920 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:08.871992 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:08.872304 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:08.872383 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:08.874644 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:08.874645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:08.874646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:08.874654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:08.876588 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:08.877088 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:08.877089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:08.877090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:08.877152 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:08.877474 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:08.877475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:08.877476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:08.877626 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:08.877705 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:08.878083 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 09:34:08.878174 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:34:08.878860 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:08.868678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42559 (local address 127.0.98.2:58941)
0926 09:34:08.868875 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:08.868917 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:08.869432 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:08.869680 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:08.869683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:08.869859 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:08.870047 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:08.870051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:08.871009 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:08.871009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:08.871897 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:08.871900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:08.872391 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:08.872392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:08.872491 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:08.873024 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:08.873033 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:08.874545 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:08.876684 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:08.876691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:08.876693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:08.876990 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:08.877231 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:08.877232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:08.877232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:08.877385 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:08.877722 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:08.877728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:08.877909 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 09:34:08.878365 (+ 456us) client_negotiation.cc:240] Negotiation successful
0926 09:34:08.878632 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":123,"threads_started":1}
I20260926 09:34:08.879786 11942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42559
I20260926 09:34:08.879889 11942 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:08.880081 11942 heartbeater.cc:511] Master 127.0.98.62:42559 requested a full tablet report, sending...
I20260926 09:34:08.880474 11592 ts_manager.cc:194] Registered new tserver with Master: a23feade64334fcdb5e6ce128abf0d0c (127.0.98.2:40067)
I20260926 09:34:08.880944 11592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58941
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:34:08.910482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42559
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:34:09.008517 11947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:09.008770 11947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:09.008849 11947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:09.010164 11947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:09.010224 11947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:09.010263 11947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:34:09.012367 11947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:09.012614 11947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:09.012876 11947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:09.019269 11952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:09.019368 11947 server_base.cc:1034] running on GCE node
W20260926 09:34:09.019414 11953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:09.019263 11955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:09.019697 11947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:09.019968 11947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:09.021131 11947 hybrid_clock.cc:648] HybridClock initialized: now 1790415249021121 us; error 36 us; skew 500 ppm
Sep 26 09:34:09 dist-test-slave-21h8 krb5kdc[11116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:09.023777 11947 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:34:09.024230 11947 webserver.cc:460] Webserver started at http://127.0.98.3:34811/ using document root <none> and password file <none>
I20260926 09:34:09.024461 11947 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:09.024528 11947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:09.024721 11947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:34:09 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:09.037518 11947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "f03e7fa24b4347819d1991daee1e0407"
format_stamp: "Formatted at 2026-09-26 09:34:09 on dist-test-slave-21h8"
server_key: "cfebcbf164a1df3037f58fc840e28c18"
server_key_iv: "d75d1d945bdb45e7f1f14774f802a0a4"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:09.038077 11947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f03e7fa24b4347819d1991daee1e0407"
format_stamp: "Formatted at 2026-09-26 09:34:09 on dist-test-slave-21h8"
server_key: "cfebcbf164a1df3037f58fc840e28c18"
server_key_iv: "d75d1d945bdb45e7f1f14774f802a0a4"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:09.040370 11947 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 09:34:09.057546 11962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:09.057803 11947 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260926 09:34:09.057898 11947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "f03e7fa24b4347819d1991daee1e0407"
format_stamp: "Formatted at 2026-09-26 09:34:09 on dist-test-slave-21h8"
server_key: "cfebcbf164a1df3037f58fc840e28c18"
server_key_iv: "d75d1d945bdb45e7f1f14774f802a0a4"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:09.057996 11947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:09.081081 11947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:09.083535 11947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:09.083765 11947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:09.084192 11947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:09.084619 11947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:09.084704 11947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:09.084779 11947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:09.084838 11947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:09.092154 11947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36803
I20260926 09:34:09.092502 12075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36803 every 8 connection(s)
I20260926 09:34:09.092660 11947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:34:09.096169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 11947
I20260926 09:34:09.096263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:34:09.096441 392 external_mini_cluster.cc:1428] Setting key e5c1e1db4e8bf51a1ddfa5e26ac8a632
Sep 26 09:34:09 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:09.108107 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.096243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41125 (local address 127.0.98.62:42559)
0926 09:34:09.096289 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:09.096291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:09.096519 (+ 228us) server_negotiation.cc:407] Connection header received
0926 09:34:09.097468 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:09.097469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:09.097480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:09.097533 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:09.098266 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.099068 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.100566 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.100869 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.100917 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.103667 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:09.103686 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:09.103687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:09.103694 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:09.105539 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:09.106050 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:09.106051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:09.106051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:09.106116 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:09.106491 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:09.106493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:09.106493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:09.106638 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:09.106705 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:09.107911 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0926 09:34:09.108007 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:34:09.108902 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.095993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42559 (local address 127.0.98.3:41125)
0926 09:34:09.096381 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:09.096432 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:09.097357 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:09.097626 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:09.097630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:09.097902 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:09.098172 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.098178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.099176 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.099178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.100454 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.100457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.101011 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.101012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.101156 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.101830 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:09.101839 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:09.103564 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:09.105635 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:09.105642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:09.105645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:09.105958 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:09.106198 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:09.106199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:09.106200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:09.106348 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:09.107520 (+ 1172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:09.107527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:09.107719 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 09:34:09.108481 (+ 762us) client_negotiation.cc:240] Negotiation successful
0926 09:34:09.108649 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":89,"threads_started":1}
I20260926 09:34:09.113830 12076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42559
I20260926 09:34:09.113951 12076 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:09.114159 12076 heartbeater.cc:511] Master 127.0.98.62:42559 requested a full tablet report, sending...
I20260926 09:34:09.114723 11591 ts_manager.cc:194] Registered new tserver with Master: f03e7fa24b4347819d1991daee1e0407 (127.0.98.3:36803)
I20260926 09:34:09.115237 11591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41125
I20260926 09:34:09.116196 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:34:09.126497 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.118089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.0.98.62:42559)
0926 09:34:09.118136 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:09.118138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:09.118381 (+ 243us) server_negotiation.cc:407] Connection header received
0926 09:34:09.118418 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:09.118418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:09.118430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:09.118500 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:09.119354 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.120122 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.121342 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.121640 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.121685 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.123345 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:09.123346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:09.123347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:09.123354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:09.125158 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:09.125599 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:09.125600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:09.125600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:09.125657 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:09.125880 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:09.125881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:09.125881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:09.126010 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:09.126086 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:09.126304 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 09:34:09.126395 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:34:09.128831 11591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:34:09.131210 11591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:34:09.145577 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.139586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36803 (local address 127.0.0.1:47936)
0926 09:34:09.140374 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:09.140417 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:09.140494 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:09.140784 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:09.140788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:09.140790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:09.140870 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.140872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.142221 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.142222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.143788 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.143792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.144937 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.144938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.145024 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.145039 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:34:09.145110 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:34:09.145478 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":105,"threads_started":1}
I20260926 09:34:09.147008 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.139670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.0.98.1:44813)
0926 09:34:09.140744 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:09.140749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:09.140758 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:34:09.140801 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:09.140807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:09.140823 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:09.140898 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:09.141218 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.142303 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.145623 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.146423 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.146495 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.146551 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:09.146901 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 09:34:09.146943 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":45,"threads_started":1}
I20260926 09:34:09.147502 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.140078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.0.98.3:36803)
0926 09:34:09.140306 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:09.140312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:09.140497 (+ 185us) server_negotiation.cc:407] Connection header received
0926 09:34:09.140582 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:09.140584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:09.140602 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:09.140688 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:09.140967 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.142118 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.143910 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.144748 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.144820 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.144898 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:09.147378 (+ 2480us) server_negotiation.cc:299] Negotiation successful
0926 09:34:09.147420 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260926 09:34:09.146827 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.139587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44813 (local address 127.0.0.1:59456)
0926 09:34:09.140607 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:09.140615 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:09.140649 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:09.140971 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:09.140972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:09.140974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:09.141019 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.141021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.142400 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.142401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.143741 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.143744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.146585 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.146586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.146650 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.146654 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:34:09.146702 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:34:09.146740 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260926 09:34:09.148276 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.139870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40067 (local address 127.0.0.1:51536)
0926 09:34:09.141112 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:09.141121 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:09.141156 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:09.144133 (+ 2977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:09.144134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:09.144136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:09.144173 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.144174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.145421 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.145422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.146819 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.146822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.148023 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.148024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.148100 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.148105 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:34:09.148160 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:34:09.148202 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"thread_start_us":65,"threads_started":1}
I20260926 09:34:09.148674 11738 tablet_service.cc:1467] Processing CreateTablet for tablet 2a667e8d74794185ac422a80cece4749 (DEFAULT_TABLE table=test-table [id=2de4d18c59174944bed77a48a0e767fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:34:09.148517 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.139999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.0.98.2:40067)
0926 09:34:09.143909 (+ 3910us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:09.143915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:09.143925 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:34:09.143963 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:09.143965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:09.143980 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:09.144051 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:09.144242 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.145322 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.146946 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.147838 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.147910 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.147985 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:09.148391 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 09:34:09.148434 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3870,"thread_start_us":107,"threads_started":1}
I20260926 09:34:09.148831 11999 tablet_service.cc:1467] Processing CreateTablet for tablet 2a667e8d74794185ac422a80cece4749 (DEFAULT_TABLE table=test-table [id=2de4d18c59174944bed77a48a0e767fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:34:09.149075 11738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a667e8d74794185ac422a80cece4749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:09.149104 11999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a667e8d74794185ac422a80cece4749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:09.152977 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 2a667e8d74794185ac422a80cece4749 (DEFAULT_TABLE table=test-table [id=2de4d18c59174944bed77a48a0e767fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:34:09.153353 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a667e8d74794185ac422a80cece4749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:09.157169 12096 tablet_bootstrap.cc:492] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: Bootstrap starting.
I20260926 09:34:09.158327 12096 tablet_bootstrap.cc:654] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:09.158667 12096 log.cc:824] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:09.159605 12096 tablet_bootstrap.cc:492] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: No bootstrap required, opened a new log
I20260926 09:34:09.159683 12096 ts_tablet_manager.cc:1397] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:34:09.161634 12096 raft_consensus.cc:348] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.161808 12096 raft_consensus.cc:374] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:09.161809 12094 tablet_bootstrap.cc:492] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355: Bootstrap starting.
I20260926 09:34:09.161850 12096 raft_consensus.cc:729] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23feade64334fcdb5e6ce128abf0d0c, State: Initialized, Role: FOLLOWER
I20260926 09:34:09.161968 12096 consensus_queue.cc:260] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.162235 12096 ts_tablet_manager.cc:1428] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:34:09.162535 12094 tablet_bootstrap.cc:654] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:09.162660 11942 heartbeater.cc:503] Master 127.0.98.62:42559 was elected leader, sending a full tablet report...
I20260926 09:34:09.162770 12095 tablet_bootstrap.cc:492] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: Bootstrap starting.
I20260926 09:34:09.164047 12094 log.cc:824] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:09.164503 12095 tablet_bootstrap.cc:654] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:09.164808 12095 log.cc:824] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:09.164970 12094 tablet_bootstrap.cc:492] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355: No bootstrap required, opened a new log
I20260926 09:34:09.165053 12094 ts_tablet_manager.cc:1397] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:34:09.167969 12095 tablet_bootstrap.cc:492] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: No bootstrap required, opened a new log
I20260926 09:34:09.168089 12095 ts_tablet_manager.cc:1397] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 09:34:09.168413 12094 raft_consensus.cc:348] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.168659 12094 raft_consensus.cc:374] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:09.168742 12094 raft_consensus.cc:729] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd6b236fd644141be5a8f52e28da355, State: Initialized, Role: FOLLOWER
I20260926 09:34:09.168977 12094 consensus_queue.cc:260] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.169339 12094 ts_tablet_manager.cc:1428] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 09:34:09.169518 11804 heartbeater.cc:503] Master 127.0.98.62:42559 was elected leader, sending a full tablet report...
I20260926 09:34:09.169909 12095 raft_consensus.cc:348] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.170212 12095 raft_consensus.cc:374] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:09.170295 12095 raft_consensus.cc:729] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e7fa24b4347819d1991daee1e0407, State: Initialized, Role: FOLLOWER
I20260926 09:34:09.170457 12095 consensus_queue.cc:260] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.170763 12095 ts_tablet_manager.cc:1428] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:34:09.170826 12076 heartbeater.cc:503] Master 127.0.98.62:42559 was elected leader, sending a full tablet report...
I20260926 09:34:09.179203 12102 raft_consensus.cc:482] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:34:09.179378 12102 raft_consensus.cc:504] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.180068 12102 leader_election.cc:288] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23feade64334fcdb5e6ce128abf0d0c (127.0.98.2:40067), 5cd6b236fd644141be5a8f52e28da355 (127.0.98.1:44813)
I20260926 09:34:09.183908 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.179914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40067 (local address 127.0.98.3:38437)
0926 09:34:09.180061 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:09.180072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:09.180117 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:09.180433 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:09.180434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:09.180435 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:09.180491 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.180492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.181359 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.181360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.182469 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.182472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.183648 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.183649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.183724 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.183728 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:34:09.183775 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:34:09.183818 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 09:34:09.183975 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.180246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41399 (local address 127.0.98.1:44813)
0926 09:34:09.180417 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:09.180418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:09.180427 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:34:09.180451 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:09.180452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:09.180464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:09.180511 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:09.180757 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.181477 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.182913 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.183567 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.183617 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.183628 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:09.183855 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 09:34:09.183905 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260926 09:34:09.184192 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.179923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44813 (local address 127.0.98.3:41399)
0926 09:34:09.180241 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:09.180252 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:09.180305 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:09.180597 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:09.180598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:09.180600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:09.180677 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.180679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.181585 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.181586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.182806 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:09.182809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.183690 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.183691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:09.183768 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.183771 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:34:09.184058 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 09:34:09.184103 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260926 09:34:09.184598 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.180196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38437 (local address 127.0.98.2:40067)
0926 09:34:09.180297 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:09.180299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:09.180307 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:34:09.180333 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:09.180333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:09.180342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:09.180384 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:09.180551 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.181243 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.182939 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.183531 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.183603 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.183616 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:09.184490 (+ 874us) server_negotiation.cc:299] Negotiation successful
0926 09:34:09.184533 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 09:34:09.184610 11758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a667e8d74794185ac422a80cece4749" candidate_uuid: "f03e7fa24b4347819d1991daee1e0407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd6b236fd644141be5a8f52e28da355" is_pre_election: true
I20260926 09:34:09.184772 11758 raft_consensus.cc:2454] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03e7fa24b4347819d1991daee1e0407 in term 0.
I20260926 09:34:09.185020 11965 leader_election.cc:302] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cd6b236fd644141be5a8f52e28da355, f03e7fa24b4347819d1991daee1e0407; no voters:
I20260926 09:34:09.185130 12102 raft_consensus.cc:2781] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:34:09.185158 12102 raft_consensus.cc:482] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:34:09.185173 12102 raft_consensus.cc:3037] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:09.185505 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a667e8d74794185ac422a80cece4749" candidate_uuid: "f03e7fa24b4347819d1991daee1e0407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23feade64334fcdb5e6ce128abf0d0c" is_pre_election: true
I20260926 09:34:09.185645 11889 raft_consensus.cc:2454] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03e7fa24b4347819d1991daee1e0407 in term 0.
I20260926 09:34:09.186139 12102 raft_consensus.cc:504] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.186293 12102 leader_election.cc:288] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [CANDIDATE]: Term 1 election: Requested vote from peers a23feade64334fcdb5e6ce128abf0d0c (127.0.98.2:40067), 5cd6b236fd644141be5a8f52e28da355 (127.0.98.1:44813)
I20260926 09:34:09.186579 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a667e8d74794185ac422a80cece4749" candidate_uuid: "f03e7fa24b4347819d1991daee1e0407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23feade64334fcdb5e6ce128abf0d0c"
I20260926 09:34:09.186591 11758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a667e8d74794185ac422a80cece4749" candidate_uuid: "f03e7fa24b4347819d1991daee1e0407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd6b236fd644141be5a8f52e28da355"
I20260926 09:34:09.186659 11889 raft_consensus.cc:3037] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:09.186663 11758 raft_consensus.cc:3037] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:09.187541 11758 raft_consensus.cc:2454] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e7fa24b4347819d1991daee1e0407 in term 1.
I20260926 09:34:09.187633 11889 raft_consensus.cc:2454] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03e7fa24b4347819d1991daee1e0407 in term 1.
I20260926 09:34:09.187703 11965 leader_election.cc:302] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cd6b236fd644141be5a8f52e28da355, f03e7fa24b4347819d1991daee1e0407; no voters:
I20260926 09:34:09.187849 12102 raft_consensus.cc:2781] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:34:09.188005 12102 raft_consensus.cc:686] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 1 LEADER]: Becoming Leader. State: Replica: f03e7fa24b4347819d1991daee1e0407, State: Running, Role: LEADER
I20260926 09:34:09.188103 12102 consensus_queue.cc:237] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } }
I20260926 09:34:09.188956 11591 catalog_manager.cc:5495] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03e7fa24b4347819d1991daee1e0407 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f03e7fa24b4347819d1991daee1e0407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03e7fa24b4347819d1991daee1e0407" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 } health_report { overall_health: UNKNOWN } } }
I20260926 09:34:09.200292 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.197114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.0.98.3:36803)
0926 09:34:09.197156 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:09.197158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:09.197167 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:34:09.197237 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:09.197238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:09.197260 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:09.197353 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:34:09.197514 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.198199 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.199340 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:09.199622 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:09.199687 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:09.199896 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:34:09.200034 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:34:09.200116 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:09.200190 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 09:34:09.200212 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:34:09.202365 11758 raft_consensus.cc:1261] T 2a667e8d74794185ac422a80cece4749 P 5cd6b236fd644141be5a8f52e28da355 [term 1 FOLLOWER]: Refusing update from remote peer f03e7fa24b4347819d1991daee1e0407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:34:09.202776 12102 consensus_queue.cc:1035] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cd6b236fd644141be5a8f52e28da355" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:34:09.203518 11889 raft_consensus.cc:1261] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 1 FOLLOWER]: Refusing update from remote peer f03e7fa24b4347819d1991daee1e0407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:34:09.203840 12102 consensus_queue.cc:1035] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23feade64334fcdb5e6ce128abf0d0c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:34:09.207806 12107 mvcc.cc:205] Tried to move back new op lower bound from 7333540860730339328 to 7333540860675207168. Current Snapshot: MvccSnapshot[applied={T|T < 7333540860730339328}]
I20260926 09:34:09.208041 12110 mvcc.cc:205] Tried to move back new op lower bound from 7333540860730339328 to 7333540860675207168. Current Snapshot: MvccSnapshot[applied={T|T < 7333540860730339328}]
I20260926 09:34:09.213439 11592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:34:09.213544 11592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:34:09.214303 11592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8a78f5d1320468a93156f2d6268104d: Sending DeleteTablet for 3 replicas of tablet 2a667e8d74794185ac422a80cece4749
I20260926 09:34:09.214929 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a667e8d74794185ac422a80cece4749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:34:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51536
I20260926 09:34:09.215410 11738 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a667e8d74794185ac422a80cece4749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:34:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59456
I20260926 09:34:09.215303 11999 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a667e8d74794185ac422a80cece4749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:34:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47936
I20260926 09:34:09.216617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 11673
Sep 26 09:34:09 dist-test-slave-21h8 krb5kdc[11116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:09.231472 12115 tablet_replica.cc:330] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: stopping tablet replica
I20260926 09:34:09.231647 12115 raft_consensus.cc:2229] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:34:09.231808 12115 raft_consensus.cc:2258] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:34:09.232062 12117 tablet_replica.cc:330] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: stopping tablet replica
I20260926 09:34:09.232188 12117 raft_consensus.cc:2229] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:34:09.232306 12117 raft_consensus.cc:2258] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:34:09.232857 12115 ts_tablet_manager.cc:1905] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:34:09.237063 12115 ts_tablet_manager.cc:1918] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:34:09.237162 12115 log.cc:1196] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/2a667e8d74794185ac422a80cece4749
I20260926 09:34:09.237774 12115 ts_tablet_manager.cc:1939] T 2a667e8d74794185ac422a80cece4749 P a23feade64334fcdb5e6ce128abf0d0c: Deleting consensus metadata
I20260926 09:34:09.238294 11579 catalog_manager.cc:4841] TS a23feade64334fcdb5e6ce128abf0d0c (127.0.98.2:40067): tablet 2a667e8d74794185ac422a80cece4749 (table test-table [id=2de4d18c59174944bed77a48a0e767fb]) successfully deleted
I20260926 09:34:09.238878 12117 ts_tablet_manager.cc:1905] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:34:09.243449 12117 ts_tablet_manager.cc:1918] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:34:09.243541 12117 log.cc:1196] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/2a667e8d74794185ac422a80cece4749
I20260926 09:34:09.243916 12117 ts_tablet_manager.cc:1939] T 2a667e8d74794185ac422a80cece4749 P f03e7fa24b4347819d1991daee1e0407: Deleting consensus metadata
I20260926 09:34:09.244449 11576 catalog_manager.cc:4841] TS f03e7fa24b4347819d1991daee1e0407 (127.0.98.3:36803): tablet 2a667e8d74794185ac422a80cece4749 (table test-table [id=2de4d18c59174944bed77a48a0e767fb]) successfully deleted
W20260926 09:34:09.257176 11965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44813 (error 108)
W20260926 09:34:09.257704 11578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44813 (error 108)
I20260926 09:34:09.259864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 11810
I20260926 09:34:09.267226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 11947
I20260926 09:34:09.279479 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:09.279191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44813 (local address 127.0.0.1:59468)
0926 09:34:09.279246 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:09.279299 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:34:09.279645 11578 catalog_manager.cc:4584] TS 5cd6b236fd644141be5a8f52e28da355 (127.0.98.1:44813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a667e8d74794185ac422a80cece4749: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44813: connect: Connection refused (error 111)
I20260926 09:34:09.284005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 11560
I20260926 09:34:09.289284 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 09:34:09.633464 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 09:34:09.633564 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 09:34:09.977809 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 09:34:09.978 UTC [11146] LOG: received smart shutdown request
2026-09-26 09:34:09.981 UTC [11146] LOG: background worker "logical replication launcher" (PID 11155) exited with exit code 1
2026-09-26 09:34:09.981 UTC [11150] LOG: shutting down
2026-09-26 09:34:10.132 UTC [11146] LOG: database system is shut down
2026-09-26T09:34:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45953 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:34:10 dist-test-slave-21h8 krb5kdc[12128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:34:10 dist-test-slave-21h8 krb5kdc[12128](info): set up 2 sockets
Sep 26 09:34:10 dist-test-slave-21h8 krb5kdc[12128](info): commencing operation
krb5kdc: starting...
W20260926 09:34:11.299719 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:34:11 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:34:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 09:34:11.461403 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5TYWI0/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/postgres -l logfile start
2026-09-26 09:34:13.864 UTC [12158] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 09:34:13.864 UTC [12158] LOG: listening on IPv4 address "127.0.98.20", port 55425
2026-09-26 09:34:13.868 UTC [12158] LOG: listening on Unix socket "/tmp/.s.PGSQL.55425"
2026-09-26 09:34:13.902 UTC [12161] LOG: database system was shut down at 2026-09-26 09:34:12 UTC
2026-09-26 09:34:13.906 UTC [12158] LOG: database system is ready to accept connections
I20260926 09:34:14.887094 392 mini_postgres.cc:96] Postgres bound to 55425
2026-09-26 09:34:14.893 UTC [12169] FATAL: database "slave" does not exist
127.0.98.20:55425 - accepting connections
I20260926 09:34:14.893503 392 mini_ranger.cc:161] Starting Ranger...
I20260926 09:34:14.907671 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 09:34:15.354254 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 09:34:15.354322 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-admin
2026-09-26 09:34:15,634 [I] DB FLAVOR :POSTGRES
2026-09-26 09:34:15,635 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:34:15,635 [I] Checking connection..
2026-09-26 09:34:15,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 09:34:15,893 [I] Checking connection passed.
2026-09-26 09:34:15,893 [I] --------- Verifying version history table ---------
2026-09-26 09:34:15,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:34:16,143 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:34:16,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:34:16,392 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:34:16,392 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 09:34:16,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 09:34:16.649 UTC [12233] WARNING: there is no transaction in progress
2026-09-26 09:34:16,655 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 09:34:16,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:34:16,880 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 09:34:16,880 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:34:16,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 09:34:17,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 09:34:17,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-21h8.c.gcp-upstream.internal','N') ;"
2026-09-26 09:34:17,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:34:17,893 [I] Table x_portal_user does not exist in database ranger
2026-09-26 09:34:17,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:34:18,151 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 09:34:18,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:34:18,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:34:18,642 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 09:34:18,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 09:34:19.563 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.579 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.594 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.608 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.623 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.640 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.703 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.710 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.722 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.732 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.741 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.754 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.763 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.768 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.774 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:19.780 UTC [12341] WARNING: there is no transaction in progress
2026-09-26 09:34:20,367 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 09:34:20,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 09:34:20,605 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 09:34:20,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:34:20,860 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 09:34:20,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:34:21,095 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 09:34:21,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:34:21,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:34:21,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-21h8.c.gcp-upstream.internal';"
2026-09-26 09:34:21,846 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 09:34:21,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 09:34:22,101 [I] DB_PATCHES have already been applied
I20260926 09:34:22.107481 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5TYWI0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 09:34:22.107555 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 09:34:22.110406 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:51335
Sep 26, 2026 9:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:34:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:34:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 09:34:23.217360 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.324s user 0.000s sys 0.008s
Sep 26, 2026 9:34:43 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:34:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 9:34:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:34:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:34:44 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 09:34:44.817951 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 09:34:44 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415284, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 09:34:44.909709 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 09:34:44.918448 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 09:34:45.692473 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 09:34:45.692541 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms
I20260926 09:34:45.692554 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:55425
2026-09-26 09:34:45,849 [I] DB FLAVOR :POSTGRES
2026-09-26 09:34:45,849 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:34:45,849 [I] Checking connection
2026-09-26 09:34:45,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 09:34:46,083 [I] connection success
2026-09-26 09:34:46,083 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 09:34:46,083 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 09:34:46,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 09:34:46,324 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 09:34:46,324 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:34:46,324 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 09:34:46,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55425/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 09:34:46,574 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 09:34:47.055678 392 mini_ranger_kms.cc:310] Created kms service
I20260926 09:34:47.261332 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 09:34:47.304916 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 09:34:47 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415284, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:47.700028 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 09:34:47.700220 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task5TYWI0/build/release/bin/postgresql.jar:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5TYWI0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 09:34:47.700242 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 09:34:47.703346 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12521
I20260926 09:34:47.703428 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:35025
Sep 26, 2026 9:34:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:34:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 09:34:49.983049 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.073s user 0.012s sys 0.128s
I20260926 09:34:49.983139 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 09:34:49.983183 392 mini_ranger_kms.cc:401] 127.0.98.20:35025/kms/v1/keys
Sep 26 09:34:50 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790415290, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 09:34:50 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415290, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 09:34:50 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415290, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 09:34:53 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:34:53.326351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33551
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33551
--ranger_config_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35025/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:34:53.430371 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:53.430555 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:53.430577 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:53.431800 12572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:34:53.431839 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:53.431854 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:53.431866 12572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:34:53.431878 12572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:34:53.433413 12572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:35025/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33551
--ranger_config_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:53.433632 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:53.433856 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:53.439447 12578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:53.439482 12572 server_base.cc:1034] running on GCE node
W20260926 09:34:53.439447 12580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:53.439630 12577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:53.439930 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:53.440222 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:53.441391 12572 hybrid_clock.cc:648] HybridClock initialized: now 1790415293441378 us; error 30 us; skew 500 ppm
Sep 26 09:34:53 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:53.443923 12572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:34:53.444418 12572 webserver.cc:460] Webserver started at http://127.0.98.62:36347/ using document root <none> and password file <none>
I20260926 09:34:53.444643 12572 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:53.444700 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:53.444830 12572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:34:53 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:53.489497 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "73054176aa20497489e65bf3ca789ee8"
format_stamp: "Formatted at 2026-09-26 09:34:53 on dist-test-slave-21h8"
server_key: "35d0cb8d02e8a324f76082e64852b9ea"
server_key_iv: "114fbf57dfd1a76ea26fcac12f721afd"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:53.489903 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "73054176aa20497489e65bf3ca789ee8"
format_stamp: "Formatted at 2026-09-26 09:34:53 on dist-test-slave-21h8"
server_key: "35d0cb8d02e8a324f76082e64852b9ea"
server_key_iv: "114fbf57dfd1a76ea26fcac12f721afd"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:53.491402 12572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:34:53.501957 12587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:53.502202 12572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:34:53.502301 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "73054176aa20497489e65bf3ca789ee8"
format_stamp: "Formatted at 2026-09-26 09:34:53 on dist-test-slave-21h8"
server_key: "35d0cb8d02e8a324f76082e64852b9ea"
server_key_iv: "114fbf57dfd1a76ea26fcac12f721afd"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:53.502416 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:53.522884 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:53.524840 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:53.524986 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:53.529800 12572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33551
I20260926 09:34:53.529835 12639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33551 every 8 connection(s)
I20260926 09:34:53.530185 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:34:53.530933 12640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:53.532557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 12572
I20260926 09:34:53.532699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:34:53.532874 392 external_mini_cluster.cc:1428] Setting key 1ffae1a728c2890edd4aa8cc627893c0
I20260926 09:34:53.535372 12640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Bootstrap starting.
I20260926 09:34:53.536099 12640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:53.536435 12640 log.cc:824] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:53.537261 12640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: No bootstrap required, opened a new log
Sep 26 09:34:53 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:53.539883 12640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260926 09:34:53.540059 12640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:53.540099 12640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73054176aa20497489e65bf3ca789ee8, State: Initialized, Role: FOLLOWER
I20260926 09:34:53.540300 12640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260926 09:34:53.540436 12640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:34:53.540472 12640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:34:53.540547 12640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:53.541245 12640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260926 09:34:53.541355 12640 leader_election.cc:302] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73054176aa20497489e65bf3ca789ee8; no voters:
I20260926 09:34:53.541514 12640 leader_election.cc:288] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:34:53.541554 12645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:34:53.541795 12645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 1 LEADER]: Becoming Leader. State: Replica: 73054176aa20497489e65bf3ca789ee8, State: Running, Role: LEADER
I20260926 09:34:53.541903 12640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:34:53.541950 12645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260926 09:34:53.542366 12646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73054176aa20497489e65bf3ca789ee8. Latest consensus state: current_term: 1 leader_uuid: "73054176aa20497489e65bf3ca789ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } } }
I20260926 09:34:53.542523 12646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:53.543418 12649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:34:53.542369 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73054176aa20497489e65bf3ca789ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } } }
I20260926 09:34:53.543730 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:53.543980 12649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:34:53.544201 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:53.533503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.0.98.62:33551)
0926 09:34:53.533798 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:53.533807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:53.533853 (+ 46us) server_negotiation.cc:407] Connection header received
0926 09:34:53.534432 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:53.534439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:53.534490 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:53.534642 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:53.535257 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:53.536327 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:53.537289 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:53.537752 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:53.537889 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:53.540270 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:53.540274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:53.540275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:53.540287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:53.542286 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:53.542731 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:53.542734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:53.542734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:53.542827 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:53.543059 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:53.543060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:53.543060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:53.543405 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:53.543527 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:53.543805 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 09:34:53.543925 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":96,"threads_started":1}
I20260926 09:34:53.544536 12640 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 09:34:53.547447 12649 catalog_manager.cc:1347] Generated new cluster ID: 24e7ee13e31d4c769d23688594560801
I20260926 09:34:53.547504 12649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:34:53.569576 12649 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:34:53.570432 12649 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:34:53.575795 12649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Generated new TSK 0
I20260926 09:34:53.576037 12649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:34:54.306998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:34:54.413244 12685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:54.413461 12685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:54.413498 12685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:54.414911 12685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:54.415011 12685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:54.415071 12685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:34:54.417222 12685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:35025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:54.417587 12685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:54.417876 12685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:54.423866 12691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:54.423911 12690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:54.424049 12693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:54.424062 12685 server_base.cc:1034] running on GCE node
I20260926 09:34:54.424391 12685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:54.424679 12685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:54.425839 12685 hybrid_clock.cc:648] HybridClock initialized: now 1790415294425828 us; error 39 us; skew 500 ppm
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:54.428752 12685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:34:54.429307 12685 webserver.cc:460] Webserver started at http://127.0.98.1:37971/ using document root <none> and password file <none>
I20260926 09:34:54.429618 12685 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:54.429751 12685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:54.429935 12685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:54.445221 12685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "59c155eea9354afbbf6cfa9e08c968db"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "f1b59c4440ca753c875a4c96198271fa"
server_key_iv: "10db2d000ac0c43deaeec8fe8214c34d"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.445670 12685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59c155eea9354afbbf6cfa9e08c968db"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "f1b59c4440ca753c875a4c96198271fa"
server_key_iv: "10db2d000ac0c43deaeec8fe8214c34d"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.452011 12685 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260926 09:34:54.467273 12702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:54.467622 12685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:34:54.467991 12685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "59c155eea9354afbbf6cfa9e08c968db"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "f1b59c4440ca753c875a4c96198271fa"
server_key_iv: "10db2d000ac0c43deaeec8fe8214c34d"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.468195 12685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:54.483491 12685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:54.485592 12685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:54.485754 12685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:54.485989 12685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:54.486379 12685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:54.486434 12685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:54.486498 12685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:54.486537 12685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:54.493618 12685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42637
I20260926 09:34:54.494011 12685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:34:54.494582 12815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42637 every 8 connection(s)
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:54.503816 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:54.494743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.1:57079)
0926 09:34:54.494909 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:54.494950 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:54.495680 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:54.495965 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:54.495970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:54.496205 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:54.496407 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:54.496412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:54.497124 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:54.497126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:54.497962 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:54.497964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:54.498384 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:54.498385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:54.498510 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:54.499021 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:54.499029 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:54.500314 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:54.502122 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:54.502128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:54.502131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:54.502422 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:54.502658 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:54.502659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:54.502659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:54.502796 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:54.503102 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:54.503108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:54.503273 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 09:34:54.503421 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 09:34:54.503570 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":99,"threads_started":1}
I20260926 09:34:54.503958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 12685
I20260926 09:34:54.504060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:34:54.504246 392 external_mini_cluster.cc:1428] Setting key db9fb66e6ae05f16ad7066bc33a85bd0
I20260926 09:34:54.504438 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:54.495049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57079 (local address 127.0.98.62:33551)
0926 09:34:54.495237 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:54.495239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:54.495250 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:34:54.495799 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:54.495800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:54.495817 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:54.495893 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:54.496507 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:54.497029 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:54.498052 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:54.498275 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:54.498362 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:54.500411 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:54.500412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:54.500413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:54.500420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:54.502035 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:54.502511 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:54.502512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:54.502513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:54.502575 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:54.502876 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:54.502877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:54.502878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:54.503013 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:54.503074 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:54.504201 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0926 09:34:54.504336 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
I20260926 09:34:54.505484 12816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260926 09:34:54.505573 12816 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:54.505750 12816 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:54.506459 12604 ts_manager.cc:194] Registered new tserver with Master: 59c155eea9354afbbf6cfa9e08c968db (127.0.98.1:42637)
I20260926 09:34:54.507354 12604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57079
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:34:54.539187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 09:34:54.632351 12824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:54.632666 12824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:54.632745 12824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:54.634440 12824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:54.634541 12824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:54.634625 12824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:34:54.636778 12824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:35025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:54.637214 12824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:54.637486 12824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:54.643608 12829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:54.644106 12832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:54.644333 12830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:54.644905 12824 server_base.cc:1034] running on GCE node
I20260926 09:34:54.645149 12824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:54.645426 12824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:54.646577 12824 hybrid_clock.cc:648] HybridClock initialized: now 1790415294646552 us; error 44 us; skew 500 ppm
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:54.649767 12824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:34:54.650296 12824 webserver.cc:460] Webserver started at http://127.0.98.2:45629/ using document root <none> and password file <none>
I20260926 09:34:54.650650 12824 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:54.650767 12824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:54.650933 12824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:54.666644 12824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b5689f82fa84561a47f4f72fa8bb83b"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "5b70c6088e5201b4a354b474559fc1ed"
server_key_iv: "80d1bf82fc653d42255e21d7055527f3"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.667160 12824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b5689f82fa84561a47f4f72fa8bb83b"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "5b70c6088e5201b4a354b474559fc1ed"
server_key_iv: "80d1bf82fc653d42255e21d7055527f3"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.668924 12824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:34:54.695289 12839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:54.695571 12824 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260926 09:34:54.695664 12824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "5b5689f82fa84561a47f4f72fa8bb83b"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "5b70c6088e5201b4a354b474559fc1ed"
server_key_iv: "80d1bf82fc653d42255e21d7055527f3"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.695760 12824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:54.730324 12824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:54.732870 12824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:54.733073 12824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:54.733357 12824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:54.733783 12824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:54.733872 12824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:54.733943 12824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:54.733999 12824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:54.740944 12824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37157
I20260926 09:34:54.741446 12824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:34:54.743903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 12824
I20260926 09:34:54.744004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:34:54.744223 392 external_mini_cluster.cc:1428] Setting key 715aec22a4782b9e897e9e5e7fb5ebc7
I20260926 09:34:54.743212 12953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37157 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:54.767696 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:54.744217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55531 (local address 127.0.98.62:33551)
0926 09:34:54.744272 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:54.744275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:54.755847 (+ 11572us) server_negotiation.cc:407] Connection header received
0926 09:34:54.757623 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:54.757625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:54.757639 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:54.757704 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:54.758559 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:54.759444 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:54.761115 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:54.761468 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:54.761525 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:54.764310 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:54.764311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:54.764312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:54.764321 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:54.765958 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:54.766517 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:54.766518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:54.766519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:54.766578 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:54.766887 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:54.766889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:54.766889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:54.767023 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:54.767064 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:54.767505 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 09:34:54.767595 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:34:54.768215 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:54.743926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.2:55531)
0926 09:34:54.755683 (+ 11757us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:54.755746 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:54.756577 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:54.757823 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:54.757830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:54.758107 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:54.758415 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:54.758422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:54.759560 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:54.759561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:54.761002 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:54.761005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:54.761652 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:54.761653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:54.761807 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:54.762417 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:54.762428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:54.764190 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:54.766055 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:54.766063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:54.766065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:54.766420 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:54.766677 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:54.766678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:54.766678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:54.766810 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:54.767110 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:54.767117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:54.767306 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 09:34:54.767801 (+ 495us) client_negotiation.cc:240] Negotiation successful
0926 09:34:54.767972 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11680,"thread_start_us":89,"threads_started":1}
I20260926 09:34:54.769016 12954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260926 09:34:54.769112 12954 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:54.769304 12954 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:54.769702 12604 ts_manager.cc:194] Registered new tserver with Master: 5b5689f82fa84561a47f4f72fa8bb83b (127.0.98.2:37157)
I20260926 09:34:54.770208 12604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:34:54.776194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:34:54.893098 12959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:54.893385 12959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:54.893421 12959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:54.895169 12959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:54.895234 12959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:54.895274 12959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:34:54.897544 12959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:35025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:54.897792 12959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:54.898108 12959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:54.904572 12967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:54.904937 12965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:54.905155 12964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:54.912803 12959 server_base.cc:1034] running on GCE node
I20260926 09:34:54.913121 12959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:54.913487 12959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:54.914680 12959 hybrid_clock.cc:648] HybridClock initialized: now 1790415294914651 us; error 44 us; skew 500 ppm
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:54.918223 12959 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:34:54.918884 12959 webserver.cc:460] Webserver started at http://127.0.98.3:33709/ using document root <none> and password file <none>
I20260926 09:34:54.919204 12959 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:54.919283 12959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:54.919553 12959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:34:54 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:54.950309 12959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "1bdba492f4c04169b38ed820a431186a"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "18c6f5fc1a248268ba616140cbbc550c"
server_key_iv: "419703ff7f599cf1cc3895e08adcdb1e"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.950766 12959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1bdba492f4c04169b38ed820a431186a"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "18c6f5fc1a248268ba616140cbbc550c"
server_key_iv: "419703ff7f599cf1cc3895e08adcdb1e"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.959509 12959 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.000s
I20260926 09:34:54.985317 12974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:54.985651 12959 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260926 09:34:54.985745 12959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "1bdba492f4c04169b38ed820a431186a"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "18c6f5fc1a248268ba616140cbbc550c"
server_key_iv: "419703ff7f599cf1cc3895e08adcdb1e"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:54.985859 12959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:55.020638 12959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:55.023343 12959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:55.023576 12959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:55.023849 12959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:55.024340 12959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:55.024432 12959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.024505 12959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:55.024564 12959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.030995 12959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35871
I20260926 09:34:55.031536 12959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:34:55.036201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 12959
I20260926 09:34:55.036298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:34:55.036481 392 external_mini_cluster.cc:1428] Setting key 32ecdfd6300ea842904b4b6ae1967f26
I20260926 09:34:55.034262 13087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35871 every 8 connection(s)
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:55.047281 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.035113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54787 (local address 127.0.98.62:33551)
0926 09:34:55.035177 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:55.035180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:55.035497 (+ 317us) server_negotiation.cc:407] Connection header received
0926 09:34:55.037591 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:55.037593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:55.037604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:55.037633 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:55.038502 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.039276 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.040769 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.041059 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.041102 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.043905 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:55.043906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:55.043906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:55.043914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:55.045671 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.046175 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.046187 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.046187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.046246 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.046532 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.046533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.046533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.046660 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:55.046702 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:55.047084 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 09:34:55.047168 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:34:55.048664 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.034837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.3:54787)
0926 09:34:55.035364 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:55.035410 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:55.037484 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:55.037824 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:55.037828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:55.038071 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:55.038392 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:55.038398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.039412 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.039413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:55.040664 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:55.040666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.041213 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.041214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:55.041344 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.041987 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:55.041998 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:55.043793 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:55.045768 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:55.045774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:55.045776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:55.046080 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:55.046320 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:55.046321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:55.046321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:55.046451 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:55.046750 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:55.046755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:55.046923 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 09:34:55.048275 (+ 1352us) client_negotiation.cc:240] Negotiation successful
0926 09:34:55.048434 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":76,"threads_started":1}
I20260926 09:34:55.049293 13088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260926 09:34:55.049384 13088 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.049571 13088 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:55.049952 12604 ts_manager.cc:194] Registered new tserver with Master: 1bdba492f4c04169b38ed820a431186a (127.0.98.3:35871)
I20260926 09:34:55.050432 12604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54787
I20260926 09:34:55.058955 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:34:55.059664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 12685
I20260926 09:34:55.067147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 12824
I20260926 09:34:55.073405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 12959
I20260926 09:34:55.080504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 12572
I20260926 09:34:55.086469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33551
--webserver_interface=127.0.98.62
--webserver_port=36347
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33551
--ranger_config_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35025/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35025/kms with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:34:55.189798 13091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:55.189996 13091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:55.190016 13091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:55.191123 13091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:34:55.191164 13091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:55.191177 13091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:55.191190 13091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:34:55.191202 13091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:34:55.192824 13091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35025/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33551
--ranger_config_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:55.193048 13091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:55.193280 13091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:55.198009 13097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:55.197974 13096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:55.198202 13099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:55.198431 13091 server_base.cc:1034] running on GCE node
I20260926 09:34:55.198691 13091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:55.199002 13091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:55.200161 13091 hybrid_clock.cc:648] HybridClock initialized: now 1790415295200137 us; error 44 us; skew 500 ppm
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:55.202888 13091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:34:55.203392 13091 webserver.cc:460] Webserver started at http://127.0.98.62:36347/ using document root <none> and password file <none>
I20260926 09:34:55.203603 13091 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:55.203683 13091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:55.265153 13091 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:34:55.266009 13107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.266211 13091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:34:55.266309 13091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "73054176aa20497489e65bf3ca789ee8"
format_stamp: "Formatted at 2026-09-26 09:34:53 on dist-test-slave-21h8"
server_key: "35d0cb8d02e8a324f76082e64852b9ea"
server_key_iv: "114fbf57dfd1a76ea26fcac12f721afd"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:55.266703 13091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:55.280220 13091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:55.281925 13091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:55.282095 13091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:55.286217 13091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33551
I20260926 09:34:55.286264 13159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33551 every 8 connection(s)
I20260926 09:34:55.286629 13091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:34:55.287822 13160 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:34:55.288493 13160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Bootstrap starting.
I20260926 09:34:55.292816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 13091
I20260926 09:34:55.293143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42637
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33551
--webserver_port=37971
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35025/kms with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:34:55.297019 13160 log.cc:824] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:55.298597 13160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:34:55.298890 13160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Bootstrap complete.
I20260926 09:34:55.301563 13160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260926 09:34:55.301798 13160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73054176aa20497489e65bf3ca789ee8, State: Initialized, Role: FOLLOWER
I20260926 09:34:55.301999 13160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260926 09:34:55.302084 13160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:34:55.302114 13160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:34:55.302181 13160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:34:55.303172 13160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260926 09:34:55.303258 13160 leader_election.cc:302] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73054176aa20497489e65bf3ca789ee8; no voters:
I20260926 09:34:55.303474 13160 leader_election.cc:288] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:34:55.303510 13164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:34:55.303611 13164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [term 2 LEADER]: Becoming Leader. State: Replica: 73054176aa20497489e65bf3ca789ee8, State: Running, Role: LEADER
I20260926 09:34:55.303730 13160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:34:55.303771 13164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } }
I20260926 09:34:55.304147 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73054176aa20497489e65bf3ca789ee8. Latest consensus state: current_term: 2 leader_uuid: "73054176aa20497489e65bf3ca789ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } } }
I20260926 09:34:55.304157 13165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73054176aa20497489e65bf3ca789ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73054176aa20497489e65bf3ca789ee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33551 } } }
I20260926 09:34:55.304339 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:55.304349 13165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:55.304562 13173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:34:55.305110 13173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:34:55.305438 13160 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 09:34:55.305871 13173 catalog_manager.cc:1259] Loaded cluster ID: 24e7ee13e31d4c769d23688594560801
I20260926 09:34:55.305922 13173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:34:55.309556 13173 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:34:55.309947 13173 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Loaded TSK: 0
I20260926 09:34:55.310215 13173 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:34:55.400341 13162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:55.400549 13162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:55.400589 13162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:55.401719 13162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:55.401785 13162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:55.401826 13162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:34:55.403249 13162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37971
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:55.403513 13162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:55.403807 13162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:55.408712 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:55.408998 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:55.409225 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:55.409507 13162 server_base.cc:1034] running on GCE node
I20260926 09:34:55.409700 13162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:55.409960 13162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:55.413944 13162 hybrid_clock.cc:648] HybridClock initialized: now 1790415295413929 us; error 30 us; skew 500 ppm
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:55.417590 13162 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:34:55.418107 13162 webserver.cc:460] Webserver started at http://127.0.98.1:37971/ using document root <none> and password file <none>
I20260926 09:34:55.418407 13162 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:55.418514 13162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:55.437821 13162 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 09:34:55.438625 13208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.438814 13162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.438891 13162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "59c155eea9354afbbf6cfa9e08c968db"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "f1b59c4440ca753c875a4c96198271fa"
server_key_iv: "10db2d000ac0c43deaeec8fe8214c34d"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:55.439292 13162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:55.471453 13162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:55.473588 13162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:55.473763 13162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:55.474015 13162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:55.474342 13162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:55.474397 13162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.474439 13162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:55.474469 13162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.481712 13162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42637
I20260926 09:34:55.481794 13321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42637 every 8 connection(s)
I20260926 09:34:55.482121 13162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:55.489117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 13162
I20260926 09:34:55.489428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37157
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33551
--webserver_port=45629
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35025/kms with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:34:55.494095 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.482920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.1:60169)
0926 09:34:55.483105 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:55.483150 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:55.483678 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:55.484135 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:55.484139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:55.484340 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:55.484521 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:55.484526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.485611 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.485612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:55.486412 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:55.486414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.487026 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.487027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:55.487155 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.487649 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:55.487658 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:55.489421 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:55.491798 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:55.491804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:55.491807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:55.492118 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:55.492542 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:55.492543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:55.492543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:55.492745 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:55.493287 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:55.493293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:55.493473 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 09:34:55.493622 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 09:34:55.493812 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260926 09:34:55.495193 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.482993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60169 (local address 127.0.98.62:33551)
0926 09:34:55.483205 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:55.483214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:55.483257 (+ 43us) server_negotiation.cc:407] Connection header received
0926 09:34:55.483915 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:55.483920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:55.483962 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:55.484096 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:55.484641 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.485505 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.486502 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.486911 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.487039 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.489543 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:55.489547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:55.489548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:55.489560 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:55.491687 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.492205 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.492208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.492208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.492401 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.492835 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.492837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.492837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.493156 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:55.493254 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:55.494807 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0926 09:34:55.494947 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260926 09:34:55.499243 13322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260926 09:34:55.499360 13322 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.499571 13322 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
W20260926 09:34:55.500548 13322 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33551 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:34:55.500627 13322 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.500703 13322 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:55.501188 13322 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.501272 13322 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
W20260926 09:34:55.501674 13322 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:34:55.590271 13326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:55.590476 13326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:55.590513 13326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:55.591951 13326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:55.592020 13326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:55.592064 13326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:34:55.593806 13326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45629
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:55.594054 13326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:55.594287 13326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:55.600514 13332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:55.600838 13334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:55.601434 13331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:55.601938 13326 server_base.cc:1034] running on GCE node
I20260926 09:34:55.602150 13326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:55.602385 13326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:55.605641 13326 hybrid_clock.cc:648] HybridClock initialized: now 1790415295605609 us; error 40 us; skew 500 ppm
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:55.609017 13326 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:34:55.609596 13326 webserver.cc:460] Webserver started at http://127.0.98.2:45629/ using document root <none> and password file <none>
I20260926 09:34:55.609872 13326 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:55.609968 13326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:55.631820 13326 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 09:34:55.633344 13342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.633564 13326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:34:55.633667 13326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "5b5689f82fa84561a47f4f72fa8bb83b"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "5b70c6088e5201b4a354b474559fc1ed"
server_key_iv: "80d1bf82fc653d42255e21d7055527f3"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:55.634158 13326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:55.682633 13326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:55.684883 13326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:55.685097 13326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:55.685374 13326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:55.685756 13326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:55.685806 13326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.685837 13326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:55.685854 13326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.693205 13326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37157
I20260926 09:34:55.693291 13455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37157 every 8 connection(s)
I20260926 09:34:55.693732 13326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:34:55.696035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 13326
I20260926 09:34:55.696327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35871
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33551
--webserver_port=33709
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35025/kms with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:55.704782 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.694501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49993 (local address 127.0.98.62:33551)
0926 09:34:55.694561 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:55.694563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:55.694807 (+ 244us) server_negotiation.cc:407] Connection header received
0926 09:34:55.695384 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:55.695386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:55.695402 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:55.695456 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:55.696113 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.696887 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.698316 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.698642 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.698714 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.701362 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:55.701363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:55.701364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:55.701371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:55.703093 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.703636 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.703637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.703637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.703701 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.703994 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.703995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.703995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.704137 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:55.704205 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:55.704586 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 09:34:55.704685 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:34:55.705526 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.694510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.2:49993)
0926 09:34:55.694691 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:55.694732 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:55.695252 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:55.695529 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:55.695533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:55.695720 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:55.695989 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:55.695996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.697008 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.697009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:55.698220 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:55.698223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.698749 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.698751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:55.698905 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.699514 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:55.699524 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:55.701241 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:55.703199 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:55.703206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:55.703208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:55.703531 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:55.703777 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:55.703777 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:55.703778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:55.703910 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:55.704242 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:55.704248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:55.704416 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 09:34:55.705125 (+ 709us) client_negotiation.cc:240] Negotiation successful
0926 09:34:55.705297 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260926 09:34:55.706280 13456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260926 09:34:55.706378 13456 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.706558 13456 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
W20260926 09:34:55.707242 13456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33551 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:34:55.707337 13456 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.707401 13456 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:55.707754 13456 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.707824 13456 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
W20260926 09:34:55.708149 13456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:34:55.812978 13459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:55.813279 13459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:55.813338 13459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:55.814962 13459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:55.815059 13459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:55.815116 13459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:34:55.816800 13459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33709
--tserver_master_addrs=127.0.98.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:55.817170 13459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:55.817466 13459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:55.824098 13466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:55.824138 13465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:55.824304 13459 server_base.cc:1034] running on GCE node
W20260926 09:34:55.824141 13468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:55.824635 13459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:55.824889 13459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:55.826056 13459 hybrid_clock.cc:648] HybridClock initialized: now 1790415295826036 us; error 39 us; skew 500 ppm
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:55.828913 13459 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:34:55.829427 13459 webserver.cc:460] Webserver started at http://127.0.98.3:33709/ using document root <none> and password file <none>
I20260926 09:34:55.829674 13459 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:55.829751 13459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:55.844049 13459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:34:55.844893 13476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.845079 13459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:34:55.845170 13459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "1bdba492f4c04169b38ed820a431186a"
format_stamp: "Formatted at 2026-09-26 09:34:54 on dist-test-slave-21h8"
server_key: "18c6f5fc1a248268ba616140cbbc550c"
server_key_iv: "419703ff7f599cf1cc3895e08adcdb1e"
server_key_version: "kuduclusterkey@0"
I20260926 09:34:55.845531 13459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:55.875540 13459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:55.878003 13459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:55.878212 13459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:55.878526 13459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:34:55.878934 13459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:34:55.878989 13459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.879034 13459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:34:55.879061 13459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:55.886282 13459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35871
I20260926 09:34:55.886472 13589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35871 every 8 connection(s)
I20260926 09:34:55.886796 13459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:34:55 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:55.895380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 13459
I20260926 09:34:55.897909 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.887647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60817 (local address 127.0.98.62:33551)
0926 09:34:55.887722 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:55.887725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:55.888355 (+ 630us) server_negotiation.cc:407] Connection header received
0926 09:34:55.888555 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:55.888556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:55.888568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:55.888633 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:55.889374 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.890162 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.891731 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.892029 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.892082 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.894630 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:55.894632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:55.894632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:55.894639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:55.896329 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.896797 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.896798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.896799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.896858 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.897164 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.897165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.897165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.897315 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:55.897355 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:55.897738 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 09:34:55.897820 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:34:55.898648 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.887603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33551 (local address 127.0.98.3:60817)
0926 09:34:55.887760 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:55.887803 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:55.888434 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:55.888703 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:55.888707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:55.888962 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:55.889256 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:55.889262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.890296 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.890297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:55.891629 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:55.891632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.892218 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.892219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:55.892359 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.893059 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:55.893070 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:55.894524 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:55.896419 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:55.896425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:55.896427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:55.896703 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:55.896932 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:55.896933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:55.896933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:55.897061 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:55.897404 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:55.897409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:55.897573 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 09:34:55.898257 (+ 684us) client_negotiation.cc:240] Negotiation successful
0926 09:34:55.898416 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260926 09:34:55.899282 13590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33551
I20260926 09:34:55.899394 13590 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.899572 13590 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
W20260926 09:34:55.900421 13590 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33551 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:34:55.900540 13590 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.900640 13590 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:55.901002 13590 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:55.901242 13590 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
W20260926 09:34:55.901582 13590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:34:55.907684 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:55.899994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.0.98.62:33551)
0926 09:34:55.900288 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:55.900289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:55.900297 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:34:55.901094 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:55.901095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:55.901106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:55.901166 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:55.901927 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.902391 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.903156 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:55.903436 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:55.903553 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:55.905052 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:55.905053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:55.905053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:55.905076 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:55.906531 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.906950 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.906951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.906951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.906999 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:55.907213 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:55.907214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:55.907214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:55.907306 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:55.907478 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:55.907569 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 09:34:55.907624 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
Sep 26 09:34:56 dist-test-slave-21h8 krb5kdc[12128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790415296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:56.501868 13322 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:56.502020 13322 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:56.502401 13124 ts_manager.cc:194] Registered new tserver with Master: 59c155eea9354afbbf6cfa9e08c968db (127.0.98.1:42637)
I20260926 09:34:56.503429 13124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60169
I20260926 09:34:56.708364 13456 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:56.708510 13456 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:56.708917 13124 ts_manager.cc:194] Registered new tserver with Master: 5b5689f82fa84561a47f4f72fa8bb83b (127.0.98.2:37157)
I20260926 09:34:56.709463 13124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49993
Sep 26 09:34:56 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:34:56.901813 13590 heartbeater.cc:464] Registering TS with master...
I20260926 09:34:56.901968 13590 heartbeater.cc:511] Master 127.0.98.62:33551 requested a full tablet report, sending...
I20260926 09:34:56.902281 13124 ts_manager.cc:194] Registered new tserver with Master: 1bdba492f4c04169b38ed820a431186a (127.0.98.3:35871)
I20260926 09:34:56.902796 13124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60817
I20260926 09:34:56.909296 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:34:56.918502 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:56.910255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.0.98.62:33551)
0926 09:34:56.910782 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:56.910784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:56.910936 (+ 152us) server_negotiation.cc:407] Connection header received
0926 09:34:56.910959 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:56.910960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:56.910971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:56.911022 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:56.911765 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.912508 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.913463 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.913741 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.913795 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:56.915276 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:56.915277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:56.915277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:56.915284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:56.917049 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:56.917454 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:56.917455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:56.917455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:56.917520 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:56.917727 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:56.917728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:56.917728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:56.917849 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:56.917928 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:56.918330 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 09:34:56.918407 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":64,"threads_started":1}
I20260926 09:34:56.920694 13124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:34:56.922883 13124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:34:56.931564 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:56.925813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35871 (local address 127.0.0.1:32872)
0926 09:34:56.927082 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:56.927091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:56.927129 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:56.927362 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:56.927362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:56.927365 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:56.927411 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:56.927412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.928653 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.928654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:56.929964 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:56.929967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.931213 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.931214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:56.931292 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:56.931306 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:34:56.931428 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 09:34:56.931485 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":149,"thread_start_us":88,"threads_started":1}
I20260926 09:34:56.931602 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:56.926287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.0.98.2:37157)
0926 09:34:56.926695 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:56.926701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:56.926711 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:34:56.926745 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:56.926747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:56.926762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:56.926828 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:56.927071 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.928219 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.929999 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.931059 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.931132 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:56.931205 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:56.931481 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 09:34:56.931524 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":68,"threads_started":1}
I20260926 09:34:56.932139 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:56.926075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.0.98.3:35871)
0926 09:34:56.926842 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:56.926846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:56.927184 (+ 338us) server_negotiation.cc:407] Connection header received
0926 09:34:56.927231 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:56.927234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:56.927249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:56.927473 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:56.927537 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.928551 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.930274 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.931016 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.931090 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:56.931165 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:56.932020 (+ 855us) server_negotiation.cc:299] Negotiation successful
0926 09:34:56.932064 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":63,"threads_started":1}
I20260926 09:34:56.932844 13524 tablet_service.cc:1467] Processing CreateTablet for tablet 250c1638ac63410492da504c47a2aa60 (DEFAULT_TABLE table=test-table [id=8ea33c4a5d394e31973faa4617f13140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:34:56.933632 13524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250c1638ac63410492da504c47a2aa60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:56.937215 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:56.925810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37157 (local address 127.0.0.1:52018)
0926 09:34:56.926471 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:56.926511 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:56.926575 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:56.926909 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:56.926912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:56.926914 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:56.926982 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:56.926984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.928604 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.928605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:56.929897 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:56.929900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.931246 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.931247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:56.931350 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:56.931355 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:34:56.931431 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:34:56.937130 (+ 5699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":5644,"thread_start_us":98,"threads_started":1}
I20260926 09:34:56.937263 13622 tablet_bootstrap.cc:492] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Bootstrap starting.
I20260926 09:34:56.937994 13622 tablet_bootstrap.cc:654] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:56.938383 13622 log.cc:824] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:56.939286 13622 tablet_bootstrap.cc:492] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: No bootstrap required, opened a new log
I20260926 09:34:56.939385 13622 ts_tablet_manager.cc:1397] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:34:56.940392 13372 tablet_service.cc:1467] Processing CreateTablet for tablet 250c1638ac63410492da504c47a2aa60 (DEFAULT_TABLE table=test-table [id=8ea33c4a5d394e31973faa4617f13140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:34:56.940716 13372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250c1638ac63410492da504c47a2aa60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:56.941254 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:56.926058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42637 (local address 127.0.0.1:43782)
0926 09:34:56.928810 (+ 2752us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:56.928821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:56.928872 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:56.937446 (+ 8574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:56.937447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:56.937450 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:56.937486 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:56.937488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.939041 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.939042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:56.940092 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:56.940095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.941036 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.941037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:56.941097 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:56.941101 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:34:56.941138 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 09:34:56.941174 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2737,"thread_start_us":81,"threads_started":1}
I20260926 09:34:56.941486 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:56.926290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.0.98.1:42637)
0926 09:34:56.926781 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:56.926786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:56.930056 (+ 3270us) server_negotiation.cc:407] Connection header received
0926 09:34:56.930094 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:56.930096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:56.930112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:56.930181 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:56.937570 (+ 7389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.938660 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.940197 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:56.940866 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:56.940934 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:56.941002 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:56.941377 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 09:34:56.941417 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":68,"threads_started":1}
I20260926 09:34:56.941992 13622 raft_consensus.cc:348] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:56.942040 13256 tablet_service.cc:1467] Processing CreateTablet for tablet 250c1638ac63410492da504c47a2aa60 (DEFAULT_TABLE table=test-table [id=8ea33c4a5d394e31973faa4617f13140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:34:56.942149 13622 raft_consensus.cc:374] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:56.942184 13622 raft_consensus.cc:729] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdba492f4c04169b38ed820a431186a, State: Initialized, Role: FOLLOWER
I20260926 09:34:56.942329 13256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250c1638ac63410492da504c47a2aa60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:56.943220 13622 consensus_queue.cc:260] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:56.943629 13622 ts_tablet_manager.cc:1428] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260926 09:34:56.943790 13590 heartbeater.cc:503] Master 127.0.98.62:33551 was elected leader, sending a full tablet report...
I20260926 09:34:56.944602 13625 tablet_bootstrap.cc:492] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Bootstrap starting.
I20260926 09:34:56.945636 13625 tablet_bootstrap.cc:654] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:56.946013 13625 log.cc:824] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:56.946473 13626 tablet_bootstrap.cc:492] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db: Bootstrap starting.
I20260926 09:34:56.946857 13625 tablet_bootstrap.cc:492] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: No bootstrap required, opened a new log
I20260926 09:34:56.946933 13625 ts_tablet_manager.cc:1397] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:34:56.947490 13626 tablet_bootstrap.cc:654] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:56.947755 13626 log.cc:824] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:56.948393 13626 tablet_bootstrap.cc:492] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db: No bootstrap required, opened a new log
I20260926 09:34:56.948454 13626 ts_tablet_manager.cc:1397] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:34:56.950884 13626 raft_consensus.cc:348] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:56.951050 13626 raft_consensus.cc:374] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:56.951086 13626 raft_consensus.cc:729] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c155eea9354afbbf6cfa9e08c968db, State: Initialized, Role: FOLLOWER
I20260926 09:34:56.951232 13626 consensus_queue.cc:260] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:56.951552 13626 ts_tablet_manager.cc:1428] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:34:56.951670 13322 heartbeater.cc:503] Master 127.0.98.62:33551 was elected leader, sending a full tablet report...
I20260926 09:34:56.951808 13625 raft_consensus.cc:348] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:56.951987 13625 raft_consensus.cc:374] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:56.952028 13625 raft_consensus.cc:729] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5689f82fa84561a47f4f72fa8bb83b, State: Initialized, Role: FOLLOWER
I20260926 09:34:56.952168 13625 consensus_queue.cc:260] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:56.952451 13625 ts_tablet_manager.cc:1428] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 09:34:56.952610 13456 heartbeater.cc:503] Master 127.0.98.62:33551 was elected leader, sending a full tablet report...
W20260926 09:34:56.983532 13323 tablet.cc:2369] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 09:34:56 dist-test-slave-21h8 krb5kdc[12128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 09:34:57.138062 13591 tablet.cc:2369] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:34:57.148075 13624 raft_consensus.cc:482] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:34:57.148241 13624 raft_consensus.cc:504] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:57.148698 13624 leader_election.cc:288] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b5689f82fa84561a47f4f72fa8bb83b (127.0.98.2:37157), 59c155eea9354afbbf6cfa9e08c968db (127.0.98.1:42637)
I20260926 09:34:57.157600 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:57.148917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56293 (local address 127.0.98.2:37157)
0926 09:34:57.148973 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:57.148975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:57.154594 (+ 5619us) server_negotiation.cc:407] Connection header received
0926 09:34:57.154696 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:57.154697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:57.154711 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:57.154740 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:57.154968 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.155636 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.156592 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.157198 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.157274 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:57.157285 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:57.157484 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 09:34:57.157526 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:34:57.157847 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:57.148863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37157 (local address 127.0.98.3:56293)
0926 09:34:57.154526 (+ 5663us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:57.154539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:57.154624 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:57.154808 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:57.154809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:57.154810 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:57.154898 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:57.154900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.155722 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.155722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:57.156490 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:57.156492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.157321 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.157322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:57.157393 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:57.157398 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:34:57.157715 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 09:34:57.157770 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5641,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260926 09:34:57.158174 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250c1638ac63410492da504c47a2aa60" candidate_uuid: "1bdba492f4c04169b38ed820a431186a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" is_pre_election: true
I20260926 09:34:57.158349 13410 raft_consensus.cc:2454] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdba492f4c04169b38ed820a431186a in term 0.
I20260926 09:34:57.158437 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:57.149293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34973 (local address 127.0.98.1:42637)
0926 09:34:57.149352 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:57.149355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:57.155090 (+ 5735us) server_negotiation.cc:407] Connection header received
0926 09:34:57.155197 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:57.155198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:57.155211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:57.155240 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:34:57.155476 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.156171 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.157499 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.158065 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.158109 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:57.158118 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:57.158333 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 09:34:57.158371 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:34:57.158586 13478 leader_election.cc:302] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdba492f4c04169b38ed820a431186a, 5b5689f82fa84561a47f4f72fa8bb83b; no voters:
I20260926 09:34:57.158735 13624 raft_consensus.cc:2781] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:34:57.158828 13624 raft_consensus.cc:482] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:34:57.158857 13624 raft_consensus.cc:3037] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:57.159852 13624 raft_consensus.cc:504] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:57.160001 13624 leader_election.cc:288] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [CANDIDATE]: Term 1 election: Requested vote from peers 5b5689f82fa84561a47f4f72fa8bb83b (127.0.98.2:37157), 59c155eea9354afbbf6cfa9e08c968db (127.0.98.1:42637)
I20260926 09:34:57.160235 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250c1638ac63410492da504c47a2aa60" candidate_uuid: "1bdba492f4c04169b38ed820a431186a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5689f82fa84561a47f4f72fa8bb83b"
I20260926 09:34:57.160324 13410 raft_consensus.cc:3037] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:57.161190 13410 raft_consensus.cc:2454] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdba492f4c04169b38ed820a431186a in term 1.
I20260926 09:34:57.161352 13478 leader_election.cc:302] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdba492f4c04169b38ed820a431186a, 5b5689f82fa84561a47f4f72fa8bb83b; no voters:
I20260926 09:34:57.161444 13624 raft_consensus.cc:2781] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:34:57.161608 13624 raft_consensus.cc:686] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 1 LEADER]: Becoming Leader. State: Replica: 1bdba492f4c04169b38ed820a431186a, State: Running, Role: LEADER
I20260926 09:34:57.161710 13624 consensus_queue.cc:237] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } }
I20260926 09:34:57.162539 13124 catalog_manager.cc:5495] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bdba492f4c04169b38ed820a431186a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1bdba492f4c04169b38ed820a431186a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdba492f4c04169b38ed820a431186a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 } health_report { overall_health: UNKNOWN } } }
I20260926 09:34:57.167476 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:57.149108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42637 (local address 127.0.98.3:34973)
0926 09:34:57.155009 (+ 5901us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:57.155022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:57.155120 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:57.155305 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:57.155305 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:57.155307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:34:57.155397 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:57.155399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.156272 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.156273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:57.157393 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:57.157395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.158193 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.158194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:57.158258 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:57.158262 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:34:57.158549 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 09:34:57.158585 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5881,"thread_start_us":79,"threads_started":1}
I20260926 09:34:57.167847 13276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250c1638ac63410492da504c47a2aa60" candidate_uuid: "1bdba492f4c04169b38ed820a431186a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c155eea9354afbbf6cfa9e08c968db" is_pre_election: true
I20260926 09:34:57.168010 13276 raft_consensus.cc:2454] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdba492f4c04169b38ed820a431186a in term 0.
I20260926 09:34:57.167866 13275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250c1638ac63410492da504c47a2aa60" candidate_uuid: "1bdba492f4c04169b38ed820a431186a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c155eea9354afbbf6cfa9e08c968db"
I20260926 09:34:57.168148 13275 raft_consensus.cc:3037] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:57.169061 13275 raft_consensus.cc:2454] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdba492f4c04169b38ed820a431186a in term 1.
W20260926 09:34:57.195003 13457 tablet.cc:2369] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:34:57.214747 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:57.212317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.0.98.3:35871)
0926 09:34:57.212452 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:57.212453 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:57.212460 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:34:57.212491 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:57.212492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:57.212511 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:57.212585 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:34:57.212717 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.213191 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.213951 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:57.214160 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:57.214238 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:57.214405 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:34:57.214512 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:34:57.214602 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:57.214642 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:34:57.214665 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 09:34:57.216771 13410 raft_consensus.cc:1261] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 1 FOLLOWER]: Refusing update from remote peer 1bdba492f4c04169b38ed820a431186a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:34:57.216755 13275 raft_consensus.cc:1261] T 250c1638ac63410492da504c47a2aa60 P 59c155eea9354afbbf6cfa9e08c968db [term 1 FOLLOWER]: Refusing update from remote peer 1bdba492f4c04169b38ed820a431186a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:34:57.217097 13635 consensus_queue.cc:1035] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5689f82fa84561a47f4f72fa8bb83b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:34:57.217273 13635 consensus_queue.cc:1035] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [LEADER]: Connected to new peer: Peer: permanent_uuid: "59c155eea9354afbbf6cfa9e08c968db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:34:57.220013 13640 mvcc.cc:205] Tried to move back new op lower bound from 7333541057397231616 to 7333541057175015424. Current Snapshot: MvccSnapshot[applied={T|T < 7333541057397231616}]
I20260926 09:34:57.220688 13638 mvcc.cc:205] Tried to move back new op lower bound from 7333541057397231616 to 7333541057175015424. Current Snapshot: MvccSnapshot[applied={T|T < 7333541057397231616}]
I20260926 09:34:57.221904 13639 mvcc.cc:205] Tried to move back new op lower bound from 7333541057397231616 to 7333541057175015424. Current Snapshot: MvccSnapshot[applied={T|T < 7333541057397231616 or (T in {7333541057397231616})}]
I20260926 09:34:57.224107 13123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:34:57.224205 13123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:34:57.224915 13123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73054176aa20497489e65bf3ca789ee8: Sending DeleteTablet for 3 replicas of tablet 250c1638ac63410492da504c47a2aa60
I20260926 09:34:57.225425 13524 tablet_service.cc:1514] Processing DeleteTablet for tablet 250c1638ac63410492da504c47a2aa60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:34:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32872
I20260926 09:34:57.225649 13372 tablet_service.cc:1514] Processing DeleteTablet for tablet 250c1638ac63410492da504c47a2aa60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:34:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52018
I20260926 09:34:57.225858 13256 tablet_service.cc:1514] Processing DeleteTablet for tablet 250c1638ac63410492da504c47a2aa60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:34:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43782
I20260926 09:34:57.226430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 13162
I20260926 09:34:57.230427 13648 tablet_replica.cc:330] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: stopping tablet replica
I20260926 09:34:57.230538 13648 raft_consensus.cc:2229] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:34:57.230652 13648 raft_consensus.cc:2258] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:34:57.234530 13647 tablet_replica.cc:330] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: stopping tablet replica
I20260926 09:34:57.234637 13647 raft_consensus.cc:2229] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:34:57.234763 13647 raft_consensus.cc:2258] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 09:34:57.235147 13108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42637 (error 108)
I20260926 09:34:57.235247 13648 ts_tablet_manager.cc:1905] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:34:57.237231 13647 ts_tablet_manager.cc:1905] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:34:57.239089 13648 ts_tablet_manager.cc:1918] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:34:57.239168 13648 log.cc:1196] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/250c1638ac63410492da504c47a2aa60
I20260926 09:34:57.239564 13647 ts_tablet_manager.cc:1918] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:34:57.239635 13647 log.cc:1196] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/250c1638ac63410492da504c47a2aa60
I20260926 09:34:57.239887 13647 ts_tablet_manager.cc:1939] T 250c1638ac63410492da504c47a2aa60 P 1bdba492f4c04169b38ed820a431186a: Deleting consensus metadata
I20260926 09:34:57.240311 13111 catalog_manager.cc:4841] TS 1bdba492f4c04169b38ed820a431186a (127.0.98.3:35871): tablet 250c1638ac63410492da504c47a2aa60 (table test-table [id=8ea33c4a5d394e31973faa4617f13140]) successfully deleted
I20260926 09:34:57.240424 13648 ts_tablet_manager.cc:1939] T 250c1638ac63410492da504c47a2aa60 P 5b5689f82fa84561a47f4f72fa8bb83b: Deleting consensus metadata
I20260926 09:34:57.240815 13109 catalog_manager.cc:4841] TS 5b5689f82fa84561a47f4f72fa8bb83b (127.0.98.2:37157): tablet 250c1638ac63410492da504c47a2aa60 (table test-table [id=8ea33c4a5d394e31973faa4617f13140]) successfully deleted
W20260926 09:34:57.243561 13616 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:34:57.241142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42637 (local address 127.0.0.1:43788)
0926 09:34:57.241201 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:57.241216 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:57.241289 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:57.243510 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 09:34:57.243721 13108 catalog_manager.cc:4584] TS 59c155eea9354afbbf6cfa9e08c968db (127.0.98.1:42637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 250c1638ac63410492da504c47a2aa60: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:34:57.247370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 13326
I20260926 09:34:57.253260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 13459
I20260926 09:34:57.258955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 13091
I20260926 09:34:57.263952 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 09:34:57.607836 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 09:34:57.607930 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 09:34:57.952327 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 09:34:57.952 UTC [12158] LOG: received smart shutdown request
2026-09-26 09:34:57.956 UTC [12158] LOG: background worker "logical replication launcher" (PID 12167) exited with exit code 1
2026-09-26 09:34:57.956 UTC [12162] LOG: shutting down
2026-09-26 09:34:58.120 UTC [12158] LOG: database system is shut down
2026-09-26T09:34:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47984 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:34:58 dist-test-slave-21h8 krb5kdc[13661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:34:58 dist-test-slave-21h8 krb5kdc[13661](info): set up 2 sockets
Sep 26 09:34:58 dist-test-slave-21h8 krb5kdc[13661](info): commencing operation
krb5kdc: starting...
W20260926 09:34:59.263154 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:34:59.382616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34115,127.0.98.61:44581,127.0.98.60:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:34:59.463798 13674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:59.463991 13674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:59.464012 13674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:59.465138 13674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:34:59.465181 13674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:59.465195 13674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:59.465250 13674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:34:59.465274 13674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:34:59.466766 13674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34115,127.0.98.61:44581,127.0.98.60:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:59.467017 13674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:59.467239 13674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:59.472088 13680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:59.472128 13674 server_base.cc:1034] running on GCE node
W20260926 09:34:59.472098 13682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:59.472095 13679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:59.472564 13674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:59.472862 13674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:59.474000 13674 hybrid_clock.cc:648] HybridClock initialized: now 1790415299473989 us; error 26 us; skew 500 ppm
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:59.476419 13674 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:34:59.476872 13674 webserver.cc:460] Webserver started at http://127.0.98.62:33819/ using document root <none> and password file <none>
I20260926 09:34:59.477046 13674 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:59.477084 13674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:59.477172 13674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:34:59.478148 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "a4d206b37745400da38e00c6e51e4b5a"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "40e577188b9b88de5aa7fb862da785d7"
server_key_iv: "4fff777e355b5b161e4e14f96caa0dde"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.478446 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4d206b37745400da38e00c6e51e4b5a"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "40e577188b9b88de5aa7fb862da785d7"
server_key_iv: "4fff777e355b5b161e4e14f96caa0dde"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.479810 13674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:34:59.480494 13689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:59.480685 13674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:34:59.480756 13674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "a4d206b37745400da38e00c6e51e4b5a"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "40e577188b9b88de5aa7fb862da785d7"
server_key_iv: "4fff777e355b5b161e4e14f96caa0dde"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.480831 13674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:59.492542 13674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:59.494340 13674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:59.494453 13674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:59.498507 13674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34115
I20260926 09:34:59.498562 13741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34115 every 8 connection(s)
I20260926 09:34:59.498904 13674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:34:59.499810 13742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:59.501295 13742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:59.507266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 13674
I20260926 09:34:59.507431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:34:59.507637 392 external_mini_cluster.cc:1428] Setting key 6acf5d32a1b1a2f4708dd1ac078daffd
I20260926 09:34:59.510788 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.501947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.0.1:44042)
0926 09:34:59.502113 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.502153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.502613 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.502861 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.502864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.503101 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.503257 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.503261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.504446 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.504447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.505160 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.505162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.505630 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.505631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.505685 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.506133 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.506141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.507687 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.509368 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.509374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.509376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.509565 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.509820 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.509821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.509821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.509908 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.510346 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.510351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.510414 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.510497 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.510626 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260926 09:34:59.510903 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.501997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.0.98.62:34115)
0926 09:34:59.502147 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.502154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.502249 (+ 95us) server_negotiation.cc:407] Connection header received
0926 09:34:59.502735 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.502737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.502753 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.502836 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.503419 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.504356 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.505234 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.505555 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.505635 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.507761 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.507787 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.507787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.507798 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.509269 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.509659 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.509661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.509662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.509743 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.509980 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.509981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.509981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.510264 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.510353 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.510552 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.510713 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":125,"thread_start_us":104,"threads_started":1}
I20260926 09:34:59.511698 13742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:34:59.512046 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.511930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37222)
0926 09:34:59.511981 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.512029 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 09:34:59.512169 13690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
I20260926 09:34:59.512562 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.512470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37228)
0926 09:34:59.512517 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.512542 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:34:59.512663 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44581: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 09:34:59.538089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44581
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34115,127.0.98.61:44581,127.0.98.60:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:34:59.561883 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } attempt: 1
I20260926 09:34:59.562409 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.562266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37244)
0926 09:34:59.562343 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.562378 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:34:59.562754 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.562657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37254)
0926 09:34:59.562701 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.562727 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:34:59.562857 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44581: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
W20260926 09:34:59.618007 13749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:59.618263 13749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:59.618310 13749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:59.619485 13749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:34:59.619557 13749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:59.619581 13749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:59.619602 13749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:34:59.619630 13749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:34:59.621147 13749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34115,127.0.98.61:44581,127.0.98.60:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:59.621443 13749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:59.621666 13749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:59.626418 13755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:59.626374 13757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:59.626394 13754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:59.626561 13749 server_base.cc:1034] running on GCE node
I20260926 09:34:59.626793 13749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:59.627138 13749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:59.628304 13749 hybrid_clock.cc:648] HybridClock initialized: now 1790415299628270 us; error 59 us; skew 500 ppm
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:59.630816 13749 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 09:34:59.631004 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } attempt: 2
I20260926 09:34:59.631412 13749 webserver.cc:460] Webserver started at http://127.0.98.61:45747/ using document root <none> and password file <none>
I20260926 09:34:59.631448 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.631267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37268)
0926 09:34:59.631377 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.631419 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 09:34:59.631668 13749 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:59.631736 13749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:59.631754 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.631662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37272)
0926 09:34:59.631708 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.631732 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:34:59.631836 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44581: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44581: connect: Connection refused (error 111)
I20260926 09:34:59.631875 13749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:34:59.632929 13749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/data/instance:
uuid: "7e74ecc10b4c4b63a094905e0b2bee20"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "6b988f7229f6919b65262a31244ddabe"
server_key_iv: "8e39fa2dbfbbdb6b18797a99a4d4937b"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.633294 13749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/wal/instance:
uuid: "7e74ecc10b4c4b63a094905e0b2bee20"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "6b988f7229f6919b65262a31244ddabe"
server_key_iv: "8e39fa2dbfbbdb6b18797a99a4d4937b"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.634572 13749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:34:59.635275 13764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:59.635483 13749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:34:59.635597 13749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/wal
uuid: "7e74ecc10b4c4b63a094905e0b2bee20"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "6b988f7229f6919b65262a31244ddabe"
server_key_iv: "8e39fa2dbfbbdb6b18797a99a4d4937b"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.635674 13749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:59.654189 13749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:59.656803 13749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:59.656986 13749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:59.661475 13749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44581
I20260926 09:34:59.661507 13816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44581 every 8 connection(s)
I20260926 09:34:59.661896 13749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:34:59.662782 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:59.663118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 13749
I20260926 09:34:59.663232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-1/wal/instance
I20260926 09:34:59.663435 392 external_mini_cluster.cc:1428] Setting key 41b2a55803dcbbb14f0c001b0e67f094
I20260926 09:34:59.665163 13817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:59.677210 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.665914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.0.98.62:34115)
0926 09:34:59.665978 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.665982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.666195 (+ 213us) server_negotiation.cc:407] Connection header received
0926 09:34:59.666683 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.666685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.666701 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.666765 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.667450 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.668283 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.669295 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.669629 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.669712 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.674296 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.674297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.674297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.674304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.675792 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.676206 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.676207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.676208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.676269 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.676558 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.676559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.676560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.676673 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.676751 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.677025 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.677111 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:34:59.677423 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.665904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.0.1:44056)
0926 09:34:59.666079 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.666113 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.666577 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.666869 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.666872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.667108 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.667300 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.667303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.668417 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.668419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.669194 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.669197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.669719 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.669720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.669817 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.670326 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.670336 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.674200 (+ 3864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.675888 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.675894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.675896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.676119 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.676347 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.676348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.676348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.676493 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.676741 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.676749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.676890 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.676994 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.677175 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260926 09:34:59.678088 13817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:34:59.686476 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.678460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.0.98.61:44581)
0926 09:34:59.678648 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.678653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.678677 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:34:59.678726 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.678728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.678741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.678815 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.679133 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.679883 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.680904 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.681182 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.681235 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.683569 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.683576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.683576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.683582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.685413 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.685770 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.685774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.685774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.685838 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.685977 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.685977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.685977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.686141 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.686217 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.686321 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.686401 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260926 09:34:59.686535 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.678386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37280)
0926 09:34:59.678604 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.678616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.678667 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.678841 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.678841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.678998 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.679058 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.679060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.679969 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.679970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.680812 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.680815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.681251 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.681252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.681309 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.681736 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.681740 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.683497 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.685494 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.685495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.685499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.685690 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.685887 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.685887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.685888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.685933 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.686199 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.686199 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.686234 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.686305 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.686445 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":46}
I20260926 09:34:59.687690 13817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:34:59.688051 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.687937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41546)
0926 09:34:59.687984 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.688033 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:34:59.688169 13765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
I20260926 09:34:59.688535 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.688457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41548)
0926 09:34:59.688489 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.688517 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 09:34:59.688670 13817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34067: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 09:34:59.690558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34067
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34115,127.0.98.61:44581,127.0.98.60:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:34:59.723084 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } attempt: 3
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:34:59.730345 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.723508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37288)
0926 09:34:59.723578 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.723592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.723642 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.723808 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.723809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.723946 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.724008 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.724010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.724960 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.724960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.725432 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.725434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.725899 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.725899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.725954 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.726488 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.726493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.727916 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.729504 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.729505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.729505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.729652 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.729839 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.729839 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.729840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.729886 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.730108 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.730109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.730143 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.730191 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.730258 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 09:34:59.730387 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.723569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.0.98.61:44581)
0926 09:34:59.723632 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.723634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.723647 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:34:59.723701 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.723702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.723714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.723788 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.724079 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.724870 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.725509 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.725814 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.725899 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.727999 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.728001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.728001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.728009 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.729424 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.729718 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.729719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.729720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.729778 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.729935 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.729935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.729936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.730055 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.730117 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.730212 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.730283 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:34:59.730966 13742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:34:59.731302 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.731198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41562)
0926 09:34:59.731245 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.731272 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:34:59.731658 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.731579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41574)
0926 09:34:59.731620 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.731643 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:34:59.731765 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34067: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
I20260926 09:34:59.737926 13817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } attempt: 1
I20260926 09:34:59.738348 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.738231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41580)
0926 09:34:59.738290 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.738324 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 09:34:59.738656 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.738599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41586)
0926 09:34:59.738621 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.738642 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 09:34:59.738755 13817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34067: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
I20260926 09:34:59.762930 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } attempt: 1
I20260926 09:34:59.763403 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.763247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41588)
0926 09:34:59.763336 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.763376 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 09:34:59.763758 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.763666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41592)
0926 09:34:59.763710 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.763737 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:34:59.763850 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34067: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34067: connect: Connection refused (error 111)
W20260926 09:34:59.774307 13824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:34:59.774518 13824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:34:59.774547 13824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:34:59.775851 13824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:34:59.775894 13824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:34:59.775910 13824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:34:59.775954 13824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:34:59.775979 13824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:34:59.777518 13824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34115,127.0.98.61:44581,127.0.98.60:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:34:59.777778 13824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:34:59.778059 13824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:34:59.782753 13832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:59.782811 13830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:34:59.782845 13829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:34:59.783215 13824 server_base.cc:1034] running on GCE node
I20260926 09:34:59.783481 13824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:34:59.783784 13824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:34:59.784976 13824 hybrid_clock.cc:648] HybridClock initialized: now 1790415299784963 us; error 44 us; skew 500 ppm
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:34:59.787684 13824 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 09:34:59.788133 13824 webserver.cc:460] Webserver started at http://127.0.98.60:40001/ using document root <none> and password file <none>
I20260926 09:34:59.788388 13824 fs_manager.cc:359] Metadata directory not provided
I20260926 09:34:59.788435 13824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:34:59.788623 13824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:34:59.789752 13824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/data/instance:
uuid: "b651a53e8ee64491aae60b4ea8218d7c"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "392045bd57e8da4514310cdea5292203"
server_key_iv: "37494d5d224cfa267cde502ebc48c79c"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.790140 13824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/wal/instance:
uuid: "b651a53e8ee64491aae60b4ea8218d7c"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "392045bd57e8da4514310cdea5292203"
server_key_iv: "37494d5d224cfa267cde502ebc48c79c"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.791667 13824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:34:59.792397 13839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:34:59.792591 13824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:34:59.792722 13824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/wal
uuid: "b651a53e8ee64491aae60b4ea8218d7c"
format_stamp: "Formatted at 2026-09-26 09:34:59 on dist-test-slave-21h8"
server_key: "392045bd57e8da4514310cdea5292203"
server_key_iv: "37494d5d224cfa267cde502ebc48c79c"
server_key_version: "encryptionkey@0"
I20260926 09:34:59.792796 13824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:34:59.801301 13824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:34:59.803036 13824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:34:59.803146 13824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:34:59.806934 13824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34067
I20260926 09:34:59.806927 13817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } attempt: 2
I20260926 09:34:59.806947 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34067 every 8 connection(s)
I20260926 09:34:59.807446 13824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:34:59.808168 13892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:34:59.809808 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:59.814823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 13824
I20260926 09:34:59.814929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790415146110679-392-0/minicluster-data/master-2/wal/instance
I20260926 09:34:59.815109 392 external_mini_cluster.cc:1428] Setting key 130a6f977dc2f06f3e1b26f48f030829
I20260926 09:34:59.816951 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.807211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41602)
0926 09:34:59.807280 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.807293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.807370 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.808588 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.808589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.808761 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.808813 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.808815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.809917 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.809918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.810582 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.810585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.811447 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.811448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.811515 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.812052 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.812061 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.813442 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.815666 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.815667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.815668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.815891 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.816153 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.816154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.816154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.816215 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.816604 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.816605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.816651 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.816739 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.816849 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 09:34:59.817155 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.807375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.0.98.60:34067)
0926 09:34:59.807906 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.807921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.807944 (+ 23us) server_negotiation.cc:407] Connection header received
0926 09:34:59.808400 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.808406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.808452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.808565 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.808922 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.809820 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.810732 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.811371 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.811495 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.813532 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.813539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.813540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.813551 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.815526 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.815990 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.815993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.815994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.816096 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.816292 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.816293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.816293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.816522 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.816610 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.816801 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.816919 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":458,"threads_started":1}
I20260926 09:34:59.818224 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.810603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.0.98.62:34115)
0926 09:34:59.810694 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.810696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.810901 (+ 205us) server_negotiation.cc:407] Connection header received
0926 09:34:59.810928 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.810929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.810940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.811000 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.811486 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.811936 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.812807 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.812992 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.813050 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.815292 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.815294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.815294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.815301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.817029 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.817432 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.817433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.817434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.817489 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.817702 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.817703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.817703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.817827 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.817867 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.818094 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.818165 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 09:34:59.818248 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.810528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34115 (local address 127.0.0.1:44060)
0926 09:34:59.810776 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.810807 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.810863 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.811020 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.811022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.811250 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.811397 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.811399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.812025 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.812026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.812701 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.812704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.813083 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.813084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.813169 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.813799 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.813803 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.815219 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.817102 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.817108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.817110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.817345 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.817559 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.817559 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.817560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.817630 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.817915 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.817921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.817970 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.818058 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.818156 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
I20260926 09:34:59.818795 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:34:59.818881 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: Bootstrap starting.
I20260926 09:34:59.819792 13817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:59.820075 13817 log.cc:824] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:59.820912 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: No bootstrap required, opened a new log
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:34:59.822891 13817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.823025 13817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:59.823091 13817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e74ecc10b4c4b63a094905e0b2bee20, State: Initialized, Role: FOLLOWER
I20260926 09:34:59.823213 13817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.823555 13817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260926 09:34:59.826220 13910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:34:59.826323 13910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:34:59.826381 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.819030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44581 (local address 127.0.0.1:37296)
0926 09:34:59.819112 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.819123 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.819167 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.819272 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.819272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.819433 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.819474 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.819475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.820137 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.820138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.820715 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.820718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.821228 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.821229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.821290 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.821713 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.821718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.823105 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.824989 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.824990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.824991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.825243 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.825671 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.825672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.825673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.825736 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.826050 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.826051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.826091 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.826154 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.826278 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 09:34:59.826720 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.819074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.0.98.61:44581)
0926 09:34:59.819115 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.819117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.819153 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:34:59.819214 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.819215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.819224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.819271 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.819552 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.820061 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.820816 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.821109 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.821155 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.823212 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.823214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.823214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.823221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.824907 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.825327 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.825328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.825328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.825595 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.825808 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.825809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.825809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.825953 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.826020 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.826520 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.826614 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:34:59.827567 13899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.827678 13899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:34:59.827733 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:34:59.825862 13749 master_runner.cc:186] Error getting master registration for 127.0.98.62:34115: OK, instance_id { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" instance_seqno: 1790415299497376 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:34:59.829133 13749 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:34:59.829627 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.815471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.0.98.60:34067)
0926 09:34:59.815620 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.815624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.815921 (+ 297us) server_negotiation.cc:407] Connection header received
0926 09:34:59.815995 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.815996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.816006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.816063 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.817035 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.820723 (+ 3688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.821548 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.821832 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.821906 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.825083 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.825084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.825085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.825090 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.828144 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.828554 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.828555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.828555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.828610 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.828798 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.828798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.828798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.828916 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.829449 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.829483 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.829541 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:34:59.835013 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.827974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41612)
0926 09:34:59.828033 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.828044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.828087 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.828224 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.828225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.828362 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.828389 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.828391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.829243 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.829244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.829996 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.829998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.830449 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.830450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.830493 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.830887 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.830893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.832263 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.834007 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.834008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.834008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.834205 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.834381 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.834381 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.834381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.834425 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.834707 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.834708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.834755 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.834834 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.834917 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:34:59.835268 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.828043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.0.98.60:34067)
0926 09:34:59.828077 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.828079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.828086 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:34:59.828153 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.828153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.828161 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.828196 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.828459 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.829158 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.830083 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.830351 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.830408 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.832348 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.832348 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.832349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.832355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.833919 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.834279 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.834279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.834280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.834323 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.834479 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.834480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.834480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.834633 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.834700 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.835109 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.835184 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:34:59.836781 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c: Bootstrap starting.
I20260926 09:34:59.837405 13892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:59.837780 13892 log.cc:824] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:59.837997 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.830288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.0.98.62:34115)
0926 09:34:59.830347 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.830348 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.830355 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:34:59.830378 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.830382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.830391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.830449 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.830927 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.831453 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.832223 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.832432 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.832506 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.834609 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.834610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.834611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.834618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.836510 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.836917 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.836919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.836919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.836976 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.837174 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.837175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.837175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.837305 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.837477 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.837834 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.837911 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:34:59.838724 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c: No bootstrap required, opened a new log
I20260926 09:34:59.838986 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } attempt: 2
I20260926 09:34:59.840935 13892 raft_consensus.cc:348] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.841084 13892 raft_consensus.cc:374] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:59.841117 13892 raft_consensus.cc:729] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b651a53e8ee64491aae60b4ea8218d7c, State: Initialized, Role: FOLLOWER
I20260926 09:34:59.841243 13892 consensus_queue.cc:260] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.841527 13892 sys_catalog.cc:564] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:34:59.841797 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.841929 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 09:34:59 dist-test-slave-21h8 krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260926 09:34:59.845013 13924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:34:59.845067 13924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:34:59.844449 13824 master_runner.cc:186] Error getting master registration for 127.0.98.62:34115: OK, instance_id { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" instance_seqno: 1790415299497376 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:34:59.847509 13824 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:34:59.847700 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.839246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34067 (local address 127.0.0.1:41624)
0926 09:34:59.839288 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 09:34:59.839297 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:34:59.839344 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:34:59.839533 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:34:59.839534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:34:59.839680 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:34:59.839715 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.839717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.840554 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.840554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.841283 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:34:59.841285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.841892 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.841893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:34:59.841952 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.842289 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:34:59.842294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:34:59.844515 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:34:59.846616 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.846617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.846617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.846828 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.847032 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:34:59.847033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:34:59.847033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:34:59.847085 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:34:59.847422 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:34:59.847424 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:34:59.847466 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 09:34:59.847515 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:34:59.847608 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 09:34:59.847859 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.838617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.0.98.61:44581)
0926 09:34:59.838669 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.838671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.838679 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:34:59.838701 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.838702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.838710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.838758 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.839469 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.840202 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.841106 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.841396 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.841452 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.844533 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.844534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.844535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.844541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.846298 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.846688 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.846689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.846690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.846738 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.846954 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.846955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.846956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.847084 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.847224 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.847659 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.847734 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:34:59.848243 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.839345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.0.98.60:34067)
0926 09:34:59.839381 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.839383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.839419 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:34:59.839439 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.839440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.839449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.839781 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.839808 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.840467 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.841473 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.841798 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.842146 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.844596 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.844597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.844597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.844603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.846526 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.846909 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.846910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.846911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.846959 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.847157 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.847158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.847158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.847279 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.848025 (+ 746us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.848078 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.848144 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:34:59.849480 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:34:59.849936 13742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: Bootstrap starting.
I20260926 09:34:59.850662 13742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: Neither blocks nor log segments found. Creating new log.
I20260926 09:34:59.850956 13742 log.cc:824] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:34:59.851827 13742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: No bootstrap required, opened a new log
I20260926 09:34:59.853188 13742 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.853325 13742 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:34:59.853370 13742 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d206b37745400da38e00c6e51e4b5a, State: Initialized, Role: FOLLOWER
I20260926 09:34:59.853494 13742 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.853755 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.853816 13742 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:34:59.853888 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:34:59.855533 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:34:59.855628 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:34:59.857802 13674 master_runner.cc:263] No leader master found from master a4d206b37745400da38e00c6e51e4b5a
I20260926 09:34:59.857865 13674 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:34:59.865868 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.858117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.0.98.62:34115)
0926 09:34:59.858152 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.858154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.858436 (+ 282us) server_negotiation.cc:407] Connection header received
0926 09:34:59.858546 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.858547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.858559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.858594 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.859399 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.860138 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.861168 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.861447 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.861488 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.862998 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.862999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.863000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.863006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.864612 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.865013 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.865014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.865014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.865067 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.865266 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.865267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.865267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.865388 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.865479 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.865684 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.865764 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:34:59.873482 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.858403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.0.98.60:34067)
0926 09:34:59.858436 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.858438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.866602 (+ 8164us) server_negotiation.cc:407] Connection header received
0926 09:34:59.866693 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.866694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.866702 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.866731 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.867394 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.868088 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.869042 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.869330 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.869371 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.870733 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.870735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.870735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.870742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.872321 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.872691 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.872692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.872692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.872742 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.872947 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.872948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.872948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.873063 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.873130 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.873318 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.873392 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:34:59.875162 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:34:59.857860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.0.98.61:44581)
0926 09:34:59.857896 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:34:59.857897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:34:59.862150 (+ 4253us) server_negotiation.cc:407] Connection header received
0926 09:34:59.863572 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:34:59.863573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:34:59.863583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:34:59.863611 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:34:59.864503 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.865228 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.869937 (+ 4709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:34:59.870249 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:34:59.870291 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:34:59.872271 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:34:59.872272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:34:59.872272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:34:59.872279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:34:59.873907 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.874290 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.874291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.874291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.874351 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:34:59.874559 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:34:59.874559 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:34:59.874560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:34:59.874676 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:34:59.874749 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:34:59.874965 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 09:34:59.875064 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 09:34:59.907681 13770 tablet.cc:2369] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:34:59.975088 13913 raft_consensus.cc:482] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:34:59.975229 13913 raft_consensus.cc:504] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.975489 13913 leader_election.cc:288] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4d206b37745400da38e00c6e51e4b5a (127.0.98.62:34115), 7e74ecc10b4c4b63a094905e0b2bee20 (127.0.98.61:44581)
I20260926 09:34:59.975845 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b651a53e8ee64491aae60b4ea8218d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" is_pre_election: true
I20260926 09:34:59.976023 13791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b651a53e8ee64491aae60b4ea8218d7c in term 0.
I20260926 09:34:59.976214 13840 leader_election.cc:302] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e74ecc10b4c4b63a094905e0b2bee20, b651a53e8ee64491aae60b4ea8218d7c; no voters:
I20260926 09:34:59.976351 13913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:34:59.976423 13913 raft_consensus.cc:482] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:34:59.976452 13913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:59.977396 13913 raft_consensus.cc:504] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.977525 13913 leader_election.cc:288] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [CANDIDATE]: Term 1 election: Requested vote from peers a4d206b37745400da38e00c6e51e4b5a (127.0.98.62:34115), 7e74ecc10b4c4b63a094905e0b2bee20 (127.0.98.61:44581)
I20260926 09:34:59.977716 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b651a53e8ee64491aae60b4ea8218d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e74ecc10b4c4b63a094905e0b2bee20"
I20260926 09:34:59.977799 13791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:59.978684 13791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b651a53e8ee64491aae60b4ea8218d7c in term 1.
I20260926 09:34:59.978864 13840 leader_election.cc:302] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e74ecc10b4c4b63a094905e0b2bee20, b651a53e8ee64491aae60b4ea8218d7c; no voters:
I20260926 09:34:59.979388 13715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b651a53e8ee64491aae60b4ea8218d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d206b37745400da38e00c6e51e4b5a"
I20260926 09:34:59.979519 13715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:34:59.980345 13715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b651a53e8ee64491aae60b4ea8218d7c in term 1.
I20260926 09:34:59.980582 13716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b651a53e8ee64491aae60b4ea8218d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d206b37745400da38e00c6e51e4b5a" is_pre_election: true
I20260926 09:34:59.980664 13716 raft_consensus.cc:2362] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b651a53e8ee64491aae60b4ea8218d7c in term 1. Re-sending same reply.
I20260926 09:34:59.980898 13913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:34:59.981055 13913 raft_consensus.cc:686] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [term 1 LEADER]: Becoming Leader. State: Replica: b651a53e8ee64491aae60b4ea8218d7c, State: Running, Role: LEADER
I20260926 09:34:59.981143 13913 consensus_queue.cc:237] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } }
I20260926 09:34:59.981494 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b651a53e8ee64491aae60b4ea8218d7c. Latest consensus state: current_term: 1 leader_uuid: "b651a53e8ee64491aae60b4ea8218d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.981566 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:59.982539 13949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:34:59.982758 13949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:34:59.984773 13791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [term 1 FOLLOWER]: Refusing update from remote peer b651a53e8ee64491aae60b4ea8218d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:34:59.985091 13913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:34:59.986266 13899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b651a53e8ee64491aae60b4ea8218d7c. Latest consensus state: current_term: 1 leader_uuid: "b651a53e8ee64491aae60b4ea8218d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.986367 13899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:34:59.986536 13716 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [term 1 FOLLOWER]: Refusing update from remote peer b651a53e8ee64491aae60b4ea8218d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:34:59.986779 13913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:34:59.988049 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b651a53e8ee64491aae60b4ea8218d7c. Latest consensus state: current_term: 1 leader_uuid: "b651a53e8ee64491aae60b4ea8218d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.988137 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:34:59.989025 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b651a53e8ee64491aae60b4ea8218d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.989116 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:59.989493 13899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b651a53e8ee64491aae60b4ea8218d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.989575 13899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:34:59.989781 13950 mvcc.cc:205] Tried to move back new op lower bound from 7333541068735549440 to 7333541068723347456. Current Snapshot: MvccSnapshot[applied={T|T < 7333541068735549440}]
I20260926 09:34:59.990293 13951 mvcc.cc:205] Tried to move back new op lower bound from 7333541068735549440 to 7333541068723347456. Current Snapshot: MvccSnapshot[applied={T|T < 7333541068735549440}]
I20260926 09:34:59.990736 13949 catalog_manager.cc:1347] Generated new cluster ID: 7e31231b98ad4f89a661839e99ed8a98
I20260926 09:34:59.990785 13949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260926 09:34:59.995138 13695 tablet.cc:2369] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:34:59.996302 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b651a53e8ee64491aae60b4ea8218d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.996382 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c [sys.catalog]: This master's current role is: LEADER
I20260926 09:34:59.997041 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b651a53e8ee64491aae60b4ea8218d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d206b37745400da38e00c6e51e4b5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34115 } } peers { permanent_uuid: "7e74ecc10b4c4b63a094905e0b2bee20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44581 } } peers { permanent_uuid: "b651a53e8ee64491aae60b4ea8218d7c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34067 } } }
I20260926 09:34:59.997118 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:35:00.010251 13949 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:35:00.011430 13949 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:35:00.017112 13949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b651a53e8ee64491aae60b4ea8218d7c: Generated new TSK 0
I20260926 09:35:00.017310 13949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:35:00.826807 13910 catalog_manager.cc:1259] Loaded cluster ID: 7e31231b98ad4f89a661839e99ed8a98
I20260926 09:35:00.826875 13910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: loading cluster ID for follower catalog manager: success
I20260926 09:35:00.827837 13910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: acquiring CA information for follower catalog manager: success
I20260926 09:35:00.828120 13910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e74ecc10b4c4b63a094905e0b2bee20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:35:00.856106 13942 catalog_manager.cc:1259] Loaded cluster ID: 7e31231b98ad4f89a661839e99ed8a98
I20260926 09:35:00.856186 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: loading cluster ID for follower catalog manager: success
I20260926 09:35:00.857028 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: acquiring CA information for follower catalog manager: success
I20260926 09:35:00.857350 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4d206b37745400da38e00c6e51e4b5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:35:01.071760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 13674
I20260926 09:35:01.076951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 13749
I20260926 09:35:01.081832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 13824
2026-09-26T09:35:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2855 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:35:01 dist-test-slave-21h8 krb5kdc[13965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:35:01 dist-test-slave-21h8 krb5kdc[13965](info): set up 2 sockets
Sep 26 09:35:01 dist-test-slave-21h8 krb5kdc[13965](info): commencing operation
krb5kdc: starting...
W20260926 09:35:02.115455 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:35:02 dist-test-slave-21h8 krb5kdc[13965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:35:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:35:02.234588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:02.318780 13978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:02.319058 13978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:02.319115 13978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:02.320367 13978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:35:02.320456 13978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:02.320483 13978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:02.320508 13978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:35:02.320541 13978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:35:02.322151 13978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:02.322458 13978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:02.322718 13978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:02.327926 13986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:02.327981 13983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:02.328051 13978 server_base.cc:1034] running on GCE node
W20260926 09:35:02.327926 13984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:02.328413 13978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:02.328666 13978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:02.329818 13978 hybrid_clock.cc:648] HybridClock initialized: now 1790415302329807 us; error 34 us; skew 500 ppm
Sep 26 09:35:02 dist-test-slave-21h8 krb5kdc[13965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:02.332505 13978 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:35:02.332953 13978 webserver.cc:460] Webserver started at http://127.0.98.62:38205/ using document root <none> and password file <none>
I20260926 09:35:02.333209 13978 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:02.333287 13978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:02.333432 13978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:02.334499 13978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "ce93812511a4437c9d56bd9333ebcf23"
format_stamp: "Formatted at 2026-09-26 09:35:02 on dist-test-slave-21h8"
server_key: "e977a2272fd699313226e108b29564d1"
server_key_iv: "d0617fc56abe8e677cec12f8b314e31e"
server_key_version: "encryptionkey@0"
I20260926 09:35:02.334859 13978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce93812511a4437c9d56bd9333ebcf23"
format_stamp: "Formatted at 2026-09-26 09:35:02 on dist-test-slave-21h8"
server_key: "e977a2272fd699313226e108b29564d1"
server_key_iv: "d0617fc56abe8e677cec12f8b314e31e"
server_key_version: "encryptionkey@0"
I20260926 09:35:02.336253 13978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:35:02.336956 13993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:02.337118 13978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:35:02.337216 13978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "ce93812511a4437c9d56bd9333ebcf23"
format_stamp: "Formatted at 2026-09-26 09:35:02 on dist-test-slave-21h8"
server_key: "e977a2272fd699313226e108b29564d1"
server_key_iv: "d0617fc56abe8e677cec12f8b314e31e"
server_key_version: "encryptionkey@0"
I20260926 09:35:02.337306 13978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:02.350829 13978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:02.353053 13978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:02.353168 13978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:02.356920 13978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39723
I20260926 09:35:02.356956 14045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39723 every 8 connection(s)
I20260926 09:35:02.357296 13978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:35:02.358162 14046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:02.359576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 13978
I20260926 09:35:02.359647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:35:02.359851 392 external_mini_cluster.cc:1428] Setting key c35d880d05fcb31b180ccb2298bf4efb
I20260926 09:35:02.360996 14046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23: Bootstrap starting.
I20260926 09:35:02.361665 14046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:02.362012 14046 log.cc:824] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:02.362744 14046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23: No bootstrap required, opened a new log
I20260926 09:35:02.365000 14046 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93812511a4437c9d56bd9333ebcf23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39723 } }
I20260926 09:35:02.365180 14046 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:02.365226 14046 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce93812511a4437c9d56bd9333ebcf23, State: Initialized, Role: FOLLOWER
I20260926 09:35:02.365353 14046 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93812511a4437c9d56bd9333ebcf23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39723 } }
I20260926 09:35:02.365434 14046 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:35:02.365465 14046 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:35:02.365506 14046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:02.366252 14046 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93812511a4437c9d56bd9333ebcf23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39723 } }
I20260926 09:35:02.366364 14046 leader_election.cc:302] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce93812511a4437c9d56bd9333ebcf23; no voters:
I20260926 09:35:02.366521 14046 leader_election.cc:288] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:35:02.366590 14051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:35:02.366777 14051 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [term 1 LEADER]: Becoming Leader. State: Replica: ce93812511a4437c9d56bd9333ebcf23, State: Running, Role: LEADER
I20260926 09:35:02.366848 14046 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:35:02.366930 14051 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93812511a4437c9d56bd9333ebcf23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39723 } }
Sep 26 09:35:02 dist-test-slave-21h8 krb5kdc[13965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:02.367305 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce93812511a4437c9d56bd9333ebcf23. Latest consensus state: current_term: 1 leader_uuid: "ce93812511a4437c9d56bd9333ebcf23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93812511a4437c9d56bd9333ebcf23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39723 } } }
I20260926 09:35:02.367340 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce93812511a4437c9d56bd9333ebcf23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce93812511a4437c9d56bd9333ebcf23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39723 } } }
I20260926 09:35:02.367408 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:02.367443 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23 [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:02.367624 14059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:35:02.368151 14059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:35:02.370014 14059 catalog_manager.cc:1347] Generated new cluster ID: 1fd5123059134644b39206a3db05e4ee
I20260926 09:35:02.370074 14059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:35:02.372377 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:02.360364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.0.98.62:39723)
0926 09:35:02.360571 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:02.360582 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:02.360618 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:35:02.361149 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:02.361158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:02.361216 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:02.361405 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:02.362009 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:02.363394 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:02.364144 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:02.364842 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:02.364958 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:02.368164 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:02.368169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:02.368170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:02.368182 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:02.370653 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:02.371073 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:02.371076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:02.371077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:02.371190 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:02.371421 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:02.371422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:02.371423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:02.371743 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:02.371909 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:02.372027 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 09:35:02.372166 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260926 09:35:02.373312 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:35:02.386390 14059 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:35:02.386988 14059 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:35:02.391943 14059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce93812511a4437c9d56bd9333ebcf23: Generated new TSK 0
I20260926 09:35:02.392117 14059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:35:02.427492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 13978
2026-09-26T09:35:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1344 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:35:02 dist-test-slave-21h8 krb5kdc[14073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:35:02 dist-test-slave-21h8 krb5kdc[14073](info): set up 2 sockets
Sep 26 09:35:02 dist-test-slave-21h8 krb5kdc[14073](info): commencing operation
krb5kdc: starting...
W20260926 09:35:03.479842 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:35:03 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:35:03.598855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:03.677510 14086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:03.677749 14086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:03.677795 14086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:03.678928 14086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:35:03.678999 14086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:03.679023 14086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:03.679046 14086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:35:03.679072 14086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:35:03.680519 14086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:03.680800 14086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:03.681036 14086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:03.685808 14094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:03.685750 14092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:03.685775 14091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:03.686030 14086 server_base.cc:1034] running on GCE node
I20260926 09:35:03.686233 14086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:03.686471 14086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:03.687608 14086 hybrid_clock.cc:648] HybridClock initialized: now 1790415303687597 us; error 27 us; skew 500 ppm
Sep 26 09:35:03 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:03.694415 14086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:35:03.694854 14086 webserver.cc:460] Webserver started at http://127.0.98.62:35907/ using document root <none> and password file <none>
I20260926 09:35:03.695075 14086 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:03.695142 14086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:03.695291 14086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:03.696426 14086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "30ea703f6d33468ab474eab36409a94c"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "cdc4797482a19fd23fe00c6aebe379cc"
server_key_iv: "235c4a55c82405151af3ebbcf4396ad7"
server_key_version: "encryptionkey@0"
I20260926 09:35:03.696781 14086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "30ea703f6d33468ab474eab36409a94c"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "cdc4797482a19fd23fe00c6aebe379cc"
server_key_iv: "235c4a55c82405151af3ebbcf4396ad7"
server_key_version: "encryptionkey@0"
I20260926 09:35:03.698051 14086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:35:03.698761 14101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:03.698942 14086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:35:03.699048 14086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "30ea703f6d33468ab474eab36409a94c"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "cdc4797482a19fd23fe00c6aebe379cc"
server_key_iv: "235c4a55c82405151af3ebbcf4396ad7"
server_key_version: "encryptionkey@0"
I20260926 09:35:03.699120 14086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:03.714718 14086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:03.716749 14086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:03.716929 14086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:03.720939 14086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38091
I20260926 09:35:03.720971 14153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38091 every 8 connection(s)
I20260926 09:35:03.721333 14086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:35:03.722255 14154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:03.723400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 14086
I20260926 09:35:03.723531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:35:03.723726 392 external_mini_cluster.cc:1428] Setting key e7ee535ea88bb5f815ca2640c1c953e6
I20260926 09:35:03.725294 14154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Bootstrap starting.
I20260926 09:35:03.725908 14154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:03.726215 14154 log.cc:824] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:03.726898 14154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: No bootstrap required, opened a new log
I20260926 09:35:03.728811 14154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:03.728988 14154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:03.729028 14154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ea703f6d33468ab474eab36409a94c, State: Initialized, Role: FOLLOWER
I20260926 09:35:03.729188 14154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:03.729257 14154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:35:03.729287 14154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:35:03.729354 14154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:03.729916 14154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:03.730036 14154 leader_election.cc:302] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ea703f6d33468ab474eab36409a94c; no voters:
I20260926 09:35:03.730227 14154 leader_election.cc:288] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:35:03.730295 14159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:35:03.730504 14159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 1 LEADER]: Becoming Leader. State: Replica: 30ea703f6d33468ab474eab36409a94c, State: Running, Role: LEADER
I20260926 09:35:03.730512 14154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:35:03.730664 14159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
Sep 26 09:35:03 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:03.730988 14159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ea703f6d33468ab474eab36409a94c. Latest consensus state: current_term: 1 leader_uuid: "30ea703f6d33468ab474eab36409a94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } } }
I20260926 09:35:03.731303 14159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:03.731209 14160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30ea703f6d33468ab474eab36409a94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } } }
I20260926 09:35:03.731798 14160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:03.731935 14168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:35:03.732441 14168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:35:03.735246 14168 catalog_manager.cc:1347] Generated new cluster ID: 0029964988e94ef7bf0679e8314d30f4
I20260926 09:35:03.735313 14168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:35:03.735581 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:03.724300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.0.98.62:38091)
0926 09:35:03.724527 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:03.724537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:03.724578 (+ 41us) server_negotiation.cc:407] Connection header received
0926 09:35:03.725102 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:03.725111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:03.725174 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:03.725337 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:03.726080 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:03.727138 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:03.727984 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:03.728406 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:03.728530 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:03.731387 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:03.731391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:03.731392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:03.731404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:03.733549 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:03.733939 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:03.733943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:03.733943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:03.734053 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:03.734277 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:03.734279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:03.734279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:03.734586 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:03.734736 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:03.735205 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 09:35:03.735375 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":117,"threads_started":1}
I20260926 09:35:03.769811 14168 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:35:03.770392 14168 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:35:03.774926 14168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Generated new TSK 0
I20260926 09:35:03.775133 14168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:35:03.812547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38091
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:03.893257 14180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:03.893488 14180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:03.893529 14180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:03.894671 14180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:03.894742 14180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:03.894794 14180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:35:03.896385 14180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:03.896658 14180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:03.896904 14180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:03.902024 14185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:03.902061 14186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:03.902209 14180 server_base.cc:1034] running on GCE node
W20260926 09:35:03.902060 14188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:03.902458 14180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:03.902704 14180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:03.903873 14180 hybrid_clock.cc:648] HybridClock initialized: now 1790415303903866 us; error 58 us; skew 500 ppm
Sep 26 09:35:03 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:03.906311 14180 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:35:03.906757 14180 webserver.cc:460] Webserver started at http://127.0.98.1:36963/ using document root <none> and password file <none>
I20260926 09:35:03.906992 14180 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:03.907058 14180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:03.907212 14180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:03.908273 14180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "c273af5c39254444ae0f2080092c18ed"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "a5c4a6b53addf91e4433d6f49ec9a3cd"
server_key_iv: "fe7743754c089f5c4dea952fa27cb6b9"
server_key_version: "encryptionkey@0"
I20260926 09:35:03.908624 14180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c273af5c39254444ae0f2080092c18ed"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "a5c4a6b53addf91e4433d6f49ec9a3cd"
server_key_iv: "fe7743754c089f5c4dea952fa27cb6b9"
server_key_version: "encryptionkey@0"
I20260926 09:35:03.909971 14180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:35:03.910738 14195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:03.910907 14180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:35:03.911003 14180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "c273af5c39254444ae0f2080092c18ed"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "a5c4a6b53addf91e4433d6f49ec9a3cd"
server_key_iv: "fe7743754c089f5c4dea952fa27cb6b9"
server_key_version: "encryptionkey@0"
I20260926 09:35:03.911082 14180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:03.921478 14180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:03.923707 14180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:03.923862 14180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:03.924073 14180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:03.924372 14180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:03.924423 14180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:03.924458 14180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:03.924486 14180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:03.930156 14180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38249
I20260926 09:35:03.930202 14308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38249 every 8 connection(s)
I20260926 09:35:03.930490 14180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:35:03 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:03.937829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 14180
I20260926 09:35:03.937937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:35:03.938120 392 external_mini_cluster.cc:1428] Setting key 8fee8c9f10f7d3346e19fcdeb4e389e7
I20260926 09:35:03.939244 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:03.931175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36531 (local address 127.0.98.62:38091)
0926 09:35:03.931231 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:03.931234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:03.931524 (+ 290us) server_negotiation.cc:407] Connection header received
0926 09:35:03.932049 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:03.932051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:03.932061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:03.932114 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:03.933034 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:03.933516 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:03.934463 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:03.934657 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:03.934707 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:03.936687 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:03.936688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:03.936688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:03.936694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:03.937847 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:03.938284 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:03.938285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:03.938285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:03.938350 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:03.938606 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:03.938607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:03.938607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:03.938716 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:03.938787 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:03.939078 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 09:35:03.939159 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:35:03.939436 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:03.931151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:36531)
0926 09:35:03.931317 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:03.931398 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:03.931961 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:03.932165 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:03.932168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:03.932766 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:03.932955 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:03.932960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:03.933597 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:03.933598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:03.934389 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:03.934391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:03.934728 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:03.934729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:03.934827 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:03.935343 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:03.935351 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:03.936603 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:03.937943 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:03.937949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:03.937951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:03.938215 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:03.938418 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:03.938419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:03.938419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:03.938536 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:03.938789 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:03.938797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:03.938937 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 09:35:03.939035 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:35:03.939187 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260926 09:35:03.939989 14309 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38091
I20260926 09:35:03.940101 14309 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:03.940275 14309 heartbeater.cc:511] Master 127.0.98.62:38091 requested a full tablet report, sending...
I20260926 09:35:03.940781 14118 ts_manager.cc:194] Registered new tserver with Master: c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249)
I20260926 09:35:03.941398 14118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36531
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:35:03.962114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38091
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:04.042336 14314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:04.042541 14314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:04.042564 14314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:04.043754 14314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:04.043797 14314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:04.043823 14314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:35:04.045238 14314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:04.045495 14314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:04.045722 14314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:35:04.050850 14314 server_base.cc:1034] running on GCE node
W20260926 09:35:04.050798 14322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:04.050828 14319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:04.050812 14320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.051144 14314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:04.051419 14314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:04.052567 14314 hybrid_clock.cc:648] HybridClock initialized: now 1790415304052556 us; error 30 us; skew 500 ppm
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:04.055028 14314 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:35:04.055538 14314 webserver.cc:460] Webserver started at http://127.0.98.2:34121/ using document root <none> and password file <none>
I20260926 09:35:04.055756 14314 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:04.055804 14314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:04.055950 14314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:04.057050 14314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "994dd2b9231e4ce3b432c340226989a3"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "2bd9eea4cbd7fe1044a7de23d87e69c8"
server_key_iv: "3e86af2e285ab8a5b57adb9a9eaa537a"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.057444 14314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "994dd2b9231e4ce3b432c340226989a3"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "2bd9eea4cbd7fe1044a7de23d87e69c8"
server_key_iv: "3e86af2e285ab8a5b57adb9a9eaa537a"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.058754 14314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:35:04.059495 14329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.059655 14314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:35:04.059743 14314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "994dd2b9231e4ce3b432c340226989a3"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "2bd9eea4cbd7fe1044a7de23d87e69c8"
server_key_iv: "3e86af2e285ab8a5b57adb9a9eaa537a"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.059818 14314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:04.094050 14314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:04.096627 14314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:04.096889 14314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:04.097182 14314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:04.097541 14314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:04.097596 14314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.097632 14314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:04.097658 14314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.103981 14314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44155
I20260926 09:35:04.104036 14442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44155 every 8 connection(s)
I20260926 09:35:04.104393 14314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:35:04.107179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 14314
I20260926 09:35:04.107309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:35:04.107535 392 external_mini_cluster.cc:1428] Setting key 01f3c48ee1fdd43a6e8df409f25443e2
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:04.114283 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.105161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52627 (local address 127.0.98.62:38091)
0926 09:35:04.105222 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.105224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.105430 (+ 206us) server_negotiation.cc:407] Connection header received
0926 09:35:04.105901 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.105902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.105911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.105973 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:04.106874 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.107534 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.108695 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.108916 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.108990 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.111236 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:04.111238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:04.111238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:04.111247 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:04.112826 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.113235 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.113237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.113237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.113309 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.113574 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.113575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.113576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.113717 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:04.113800 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.114081 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.114178 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:35:04.114444 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.105143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:52627)
0926 09:35:04.105303 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.105346 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.105822 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.106028 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.106031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.106596 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:04.106798 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.106803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.107635 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.107637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.108591 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.108593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.109011 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.109012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.109137 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.109706 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:04.109715 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:04.111139 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:04.112911 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.112918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.112920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.113154 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.113378 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.113379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.113380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.113498 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.113791 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:04.113797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:04.113941 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.114047 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.114207 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260926 09:35:04.114925 14443 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38091
I20260926 09:35:04.115020 14443 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:04.115221 14443 heartbeater.cc:511] Master 127.0.98.62:38091 requested a full tablet report, sending...
I20260926 09:35:04.115658 14118 ts_manager.cc:194] Registered new tserver with Master: 994dd2b9231e4ce3b432c340226989a3 (127.0.98.2:44155)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:35:04.116154 14118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:35:04.133956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38091
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:04.213208 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:04.213477 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:04.213502 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:04.214854 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:04.214906 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:04.214934 14448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:35:04.216399 14448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:04.216588 14448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:04.216766 14448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:04.222059 14456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:04.222076 14453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.222105 14448 server_base.cc:1034] running on GCE node
W20260926 09:35:04.222069 14454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.222546 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:04.222820 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:04.224022 14448 hybrid_clock.cc:648] HybridClock initialized: now 1790415304224005 us; error 48 us; skew 500 ppm
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:04.226506 14448 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:35:04.227023 14448 webserver.cc:460] Webserver started at http://127.0.98.3:40147/ using document root <none> and password file <none>
I20260926 09:35:04.227276 14448 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:04.227367 14448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:04.227514 14448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:04.228585 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "a68137e848ee4643b02ee70de19c9dcc"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "57acbd6fd7eb3fb1f5611a734f24437f"
server_key_iv: "610c8f1c0bf9f11e689da7a8d4645fe4"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.228953 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a68137e848ee4643b02ee70de19c9dcc"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "57acbd6fd7eb3fb1f5611a734f24437f"
server_key_iv: "610c8f1c0bf9f11e689da7a8d4645fe4"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.230352 14448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:35:04.231170 14463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.231427 14448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:04.231529 14448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "a68137e848ee4643b02ee70de19c9dcc"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "57acbd6fd7eb3fb1f5611a734f24437f"
server_key_iv: "610c8f1c0bf9f11e689da7a8d4645fe4"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.231619 14448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:04.244866 14448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:04.246932 14448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:04.247121 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:04.247486 14448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:04.247797 14448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:04.247833 14448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.247857 14448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:04.247870 14448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.254005 14448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37907
I20260926 09:35:04.254087 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37907 every 8 connection(s)
I20260926 09:35:04.254433 14448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:35:04.258476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 14448
I20260926 09:35:04.258581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:35:04.258749 392 external_mini_cluster.cc:1428] Setting key 7d869745fdc1159bdf4b3059650e6955
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:04.264061 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.255227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57731 (local address 127.0.98.62:38091)
0926 09:35:04.255335 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.255338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.255551 (+ 213us) server_negotiation.cc:407] Connection header received
0926 09:35:04.256121 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.256123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.256134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.256195 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:04.257161 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.257654 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.258923 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.259200 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.259266 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.261466 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:04.261467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:04.261468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:04.261474 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:04.262735 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.263131 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.263132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.263133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.263186 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.263476 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.263490 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.263490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.263618 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:04.263674 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.263931 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.263995 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 09:35:04.264206 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.255188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:57731)
0926 09:35:04.255407 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.255454 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.256003 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.256271 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.256274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.256871 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:04.257076 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.257080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.257734 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.257735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.258799 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.258802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.259291 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.259292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.259476 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.260019 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:04.260028 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:04.261385 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:04.262806 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.262812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.262814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.263043 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.263252 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.263252 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.263253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.263427 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.263679 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:04.263684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:04.263801 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.263890 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.264012 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":137,"threads_started":1}
I20260926 09:35:04.264657 14577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38091
I20260926 09:35:04.264761 14577 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:04.264966 14577 heartbeater.cc:511] Master 127.0.98.62:38091 requested a full tablet report, sending...
I20260926 09:35:04.265342 14118 ts_manager.cc:194] Registered new tserver with Master: a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907)
I20260926 09:35:04.265709 14118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57731
I20260926 09:35:04.270687 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:35:04.277586 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.272047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.0.98.62:38091)
0926 09:35:04.272119 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.272121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.272168 (+ 47us) server_negotiation.cc:407] Connection header received
0926 09:35:04.272222 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.272223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.272231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.272278 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:04.272796 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.273254 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.273948 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.274117 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.274165 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.275500 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:04.275501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:04.275501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:04.275507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:04.276632 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.276929 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.276929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.276930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.276978 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.277134 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.277135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.277135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.277266 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:04.277352 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.277445 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.277499 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:35:04.279237 14118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:35:04.280879 14118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:35:04.287497 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.283204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.0.98.3:37907)
0926 09:35:04.283679 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.283684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.283693 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:35:04.283728 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.283730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.283744 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.283838 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:04.284043 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.284852 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.286558 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.287057 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.287146 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.287206 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.287359 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.287402 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":84,"threads_started":1}
I20260926 09:35:04.287526 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.283089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44155 (local address 127.0.0.1:55586)
0926 09:35:04.283252 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.283294 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.283382 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.283717 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.283721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.283722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:04.283789 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.283791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.285126 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.285127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.286282 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.286284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.287232 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.287233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.287307 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.287331 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.287386 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.287443 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":16,"thread_start_us":60,"threads_started":1}
I20260926 09:35:04.287629 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.283531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.0.98.1:38249)
0926 09:35:04.283706 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.283711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.284076 (+ 365us) server_negotiation.cc:407] Connection header received
0926 09:35:04.284121 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.284123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.284138 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.284215 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:04.284366 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.285337 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.286308 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.287059 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.287132 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.287193 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.287491 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.287535 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260926 09:35:04.287827 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.283307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.0.98.2:44155)
0926 09:35:04.283463 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.283468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.283481 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:35:04.283525 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.283528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.283544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.283622 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:04.283873 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.285024 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.286349 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.287059 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.287132 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.287193 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.287711 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.287765 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260926 09:35:04.288015 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.283114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37907 (local address 127.0.0.1:58534)
0926 09:35:04.283485 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.283497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.283550 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.283896 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.283897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.283899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:04.283956 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.283958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.284954 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.284955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.286474 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.286475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.287147 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.287148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.287229 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.287243 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.287352 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.287411 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":5,"thread_start_us":85,"threads_started":1}
I20260926 09:35:04.288029 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.283291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.0.1:60428)
0926 09:35:04.283954 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.283964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.284013 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.284232 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.284233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.284235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:04.284279 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.284280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.285437 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.285438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.286195 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.286197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.287234 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.287235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.287299 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.287304 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.287386 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.287443 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":26,"thread_start_us":106,"threads_started":1}
I20260926 09:35:04.288558 14377 tablet_service.cc:1467] Processing CreateTablet for tablet df4548b9bcaf451585de4f5202924442 (DEFAULT_TABLE table=test-table [id=e44da20ad3ac478d9e68830b5ef809c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:04.288782 14511 tablet_service.cc:1467] Processing CreateTablet for tablet df4548b9bcaf451585de4f5202924442 (DEFAULT_TABLE table=test-table [id=e44da20ad3ac478d9e68830b5ef809c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:04.288947 14377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4548b9bcaf451585de4f5202924442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:04.288913 14243 tablet_service.cc:1467] Processing CreateTablet for tablet df4548b9bcaf451585de4f5202924442 (DEFAULT_TABLE table=test-table [id=e44da20ad3ac478d9e68830b5ef809c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:04.289114 14511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4548b9bcaf451585de4f5202924442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:04.289213 14243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4548b9bcaf451585de4f5202924442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:04.291851 14594 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: Bootstrap starting.
I20260926 09:35:04.292464 14596 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Bootstrap starting.
I20260926 09:35:04.292588 14594 tablet_bootstrap.cc:654] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:04.293056 14596 tablet_bootstrap.cc:654] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:04.293125 14594 log.cc:824] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:04.293649 14596 log.cc:824] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:04.293973 14594 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: No bootstrap required, opened a new log
I20260926 09:35:04.293987 14595 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Bootstrap starting.
I20260926 09:35:04.294054 14594 ts_tablet_manager.cc:1397] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:04.294523 14595 tablet_bootstrap.cc:654] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:04.294983 14595 log.cc:824] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:04.295534 14596 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: No bootstrap required, opened a new log
I20260926 09:35:04.295606 14596 ts_tablet_manager.cc:1397] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:35:04.295871 14594 raft_consensus.cc:348] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.296036 14594 raft_consensus.cc:374] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:04.296093 14594 raft_consensus.cc:729] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68137e848ee4643b02ee70de19c9dcc, State: Initialized, Role: FOLLOWER
I20260926 09:35:04.295769 14595 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: No bootstrap required, opened a new log
I20260926 09:35:04.296176 14595 ts_tablet_manager.cc:1397] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:04.296223 14594 consensus_queue.cc:260] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.296542 14594 ts_tablet_manager.cc:1428] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:04.296548 14577 heartbeater.cc:503] Master 127.0.98.62:38091 was elected leader, sending a full tablet report...
I20260926 09:35:04.297668 14596 raft_consensus.cc:348] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.297852 14596 raft_consensus.cc:374] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:04.297888 14596 raft_consensus.cc:729] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c273af5c39254444ae0f2080092c18ed, State: Initialized, Role: FOLLOWER
I20260926 09:35:04.298003 14596 consensus_queue.cc:260] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.298280 14596 ts_tablet_manager.cc:1428] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:35:04.298305 14309 heartbeater.cc:503] Master 127.0.98.62:38091 was elected leader, sending a full tablet report...
I20260926 09:35:04.298220 14595 raft_consensus.cc:348] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.298388 14595 raft_consensus.cc:374] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:04.298425 14595 raft_consensus.cc:729] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994dd2b9231e4ce3b432c340226989a3, State: Initialized, Role: FOLLOWER
I20260926 09:35:04.298537 14595 consensus_queue.cc:260] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.298816 14595 ts_tablet_manager.cc:1428] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:35:04.298838 14443 heartbeater.cc:503] Master 127.0.98.62:38091 was elected leader, sending a full tablet report...
I20260926 09:35:04.350903 14601 raft_consensus.cc:482] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:35:04.351032 14601 raft_consensus.cc:504] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.351562 14601 leader_election.cc:288] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 994dd2b9231e4ce3b432c340226989a3 (127.0.98.2:44155), a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907)
I20260926 09:35:04.354266 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.351793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40077 (local address 127.0.98.2:44155)
0926 09:35:04.351847 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.351849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.351860 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:35:04.351925 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.351926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.351936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.351993 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:04.352147 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.352627 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.353541 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.353915 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.354010 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.354020 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.354163 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.354200 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:35:04.354266 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.351740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44155 (local address 127.0.98.1:40077)
0926 09:35:04.351810 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.351821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.351868 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.351997 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.351998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.351999 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:04.352078 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.352080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.352709 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.352711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.353448 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.353450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.354009 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.354010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.354089 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.354092 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.354158 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.354197 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 09:35:04.354439 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.351740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37907 (local address 127.0.98.1:57163)
0926 09:35:04.351941 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.351950 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.352037 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.352184 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.352185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.352186 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:04.352255 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.352256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.352870 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.352871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.353601 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.353603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.354169 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.354170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.354240 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.354243 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.354318 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.354360 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":13,"thread_start_us":122,"threads_started":1}
I20260926 09:35:04.354424 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.351829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57163 (local address 127.0.98.3:37907)
0926 09:35:04.351878 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.351880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.352023 (+ 143us) server_negotiation.cc:407] Connection header received
0926 09:35:04.352093 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.352094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.352107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.352180 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:04.352329 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.352779 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.353708 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.354084 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.354166 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.354175 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.354322 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.354363 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:35:04.354628 14397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4548b9bcaf451585de4f5202924442" candidate_uuid: "c273af5c39254444ae0f2080092c18ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994dd2b9231e4ce3b432c340226989a3" is_pre_election: true
I20260926 09:35:04.354748 14397 raft_consensus.cc:2454] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c273af5c39254444ae0f2080092c18ed in term 0.
W20260926 09:35:04.354954 14444 tablet.cc:2369] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:35:04.354975 14196 leader_election.cc:302] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994dd2b9231e4ce3b432c340226989a3, c273af5c39254444ae0f2080092c18ed; no voters:
I20260926 09:35:04.355096 14601 raft_consensus.cc:2781] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:35:04.355149 14601 raft_consensus.cc:482] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:35:04.355177 14601 raft_consensus.cc:3037] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:04.355137 14531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4548b9bcaf451585de4f5202924442" candidate_uuid: "c273af5c39254444ae0f2080092c18ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68137e848ee4643b02ee70de19c9dcc" is_pre_election: true
I20260926 09:35:04.355276 14531 raft_consensus.cc:2454] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c273af5c39254444ae0f2080092c18ed in term 0.
I20260926 09:35:04.355970 14601 raft_consensus.cc:504] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.356102 14601 leader_election.cc:288] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [CANDIDATE]: Term 1 election: Requested vote from peers 994dd2b9231e4ce3b432c340226989a3 (127.0.98.2:44155), a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907)
I20260926 09:35:04.356237 14531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4548b9bcaf451585de4f5202924442" candidate_uuid: "c273af5c39254444ae0f2080092c18ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68137e848ee4643b02ee70de19c9dcc"
I20260926 09:35:04.356283 14397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4548b9bcaf451585de4f5202924442" candidate_uuid: "c273af5c39254444ae0f2080092c18ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994dd2b9231e4ce3b432c340226989a3"
I20260926 09:35:04.356323 14531 raft_consensus.cc:3037] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:04.356407 14397 raft_consensus.cc:3037] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:04.357259 14397 raft_consensus.cc:2454] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c273af5c39254444ae0f2080092c18ed in term 1.
I20260926 09:35:04.357259 14531 raft_consensus.cc:2454] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c273af5c39254444ae0f2080092c18ed in term 1.
I20260926 09:35:04.357415 14199 leader_election.cc:302] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a68137e848ee4643b02ee70de19c9dcc, c273af5c39254444ae0f2080092c18ed; no voters:
I20260926 09:35:04.357498 14601 raft_consensus.cc:2781] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:35:04.357645 14601 raft_consensus.cc:686] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 LEADER]: Becoming Leader. State: Replica: c273af5c39254444ae0f2080092c18ed, State: Running, Role: LEADER
I20260926 09:35:04.357757 14601 consensus_queue.cc:237] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.358631 14117 catalog_manager.cc:5495] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c273af5c39254444ae0f2080092c18ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c273af5c39254444ae0f2080092c18ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } health_report { overall_health: HEALTHY } } }
I20260926 09:35:04.379897 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.377556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.0.98.1:38249)
0926 09:35:04.377603 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.377606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.377615 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:35:04.377669 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.377670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.377689 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.377755 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:35:04.377898 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.378350 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.379107 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.379353 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.379413 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.379579 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:35:04.379679 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:35:04.379766 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.379800 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.379819 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:35:04.381506 14397 raft_consensus.cc:1261] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Refusing update from remote peer c273af5c39254444ae0f2080092c18ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:35:04.381546 14531 raft_consensus.cc:1261] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 1 FOLLOWER]: Refusing update from remote peer c273af5c39254444ae0f2080092c18ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:35:04.381876 14601 consensus_queue.cc:1035] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:35:04.382067 14604 consensus_queue.cc:1035] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:35:04.384497 14609 mvcc.cc:205] Tried to move back new op lower bound from 7333541086744190976 to 7333541086649942016. Current Snapshot: MvccSnapshot[applied={T|T < 7333541086744190976}]
I20260926 09:35:04.384958 14608 mvcc.cc:205] Tried to move back new op lower bound from 7333541086744190976 to 7333541086649942016. Current Snapshot: MvccSnapshot[applied={T|T < 7333541086744190976}]
I20260926 09:35:04.386488 14118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:35:04.386596 14118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:35:04.387388 14118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Sending DeleteTablet for 3 replicas of tablet df4548b9bcaf451585de4f5202924442
I20260926 09:35:04.387744 14511 tablet_service.cc:1514] Processing DeleteTablet for tablet df4548b9bcaf451585de4f5202924442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58534
I20260926 09:35:04.387733 14377 tablet_service.cc:1514] Processing DeleteTablet for tablet df4548b9bcaf451585de4f5202924442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55586
I20260926 09:35:04.387820 14243 tablet_service.cc:1514] Processing DeleteTablet for tablet df4548b9bcaf451585de4f5202924442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60428
I20260926 09:35:04.387974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 14180
I20260926 09:35:04.388008 14619 tablet_replica.cc:330] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: stopping tablet replica
I20260926 09:35:04.388149 14619 raft_consensus.cc:2229] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:35:04.388342 14619 raft_consensus.cc:2258] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:04.389158 14618 tablet_replica.cc:330] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: stopping tablet replica
I20260926 09:35:04.389266 14618 raft_consensus.cc:2229] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:04.389366 14618 raft_consensus.cc:2258] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:04.389537 14617 tablet_replica.cc:330] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: stopping tablet replica
I20260926 09:35:04.389631 14617 raft_consensus.cc:2229] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:04.389735 14617 raft_consensus.cc:2258] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:04.389830 14618 ts_tablet_manager.cc:1905] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:04.390342 14617 ts_tablet_manager.cc:1905] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:04.391871 14618 ts_tablet_manager.cc:1918] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:04.391935 14617 ts_tablet_manager.cc:1918] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:04.391983 14618 log.cc:1196] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/df4548b9bcaf451585de4f5202924442
I20260926 09:35:04.392057 14617 log.cc:1196] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/df4548b9bcaf451585de4f5202924442
I20260926 09:35:04.392249 14618 ts_tablet_manager.cc:1939] T df4548b9bcaf451585de4f5202924442 P 994dd2b9231e4ce3b432c340226989a3: Deleting consensus metadata
I20260926 09:35:04.392395 14617 ts_tablet_manager.cc:1939] T df4548b9bcaf451585de4f5202924442 P a68137e848ee4643b02ee70de19c9dcc: Deleting consensus metadata
I20260926 09:35:04.392660 14102 catalog_manager.cc:4841] TS 994dd2b9231e4ce3b432c340226989a3 (127.0.98.2:44155): tablet df4548b9bcaf451585de4f5202924442 (table test-table [id=e44da20ad3ac478d9e68830b5ef809c5]) successfully deleted
I20260926 09:35:04.392840 14105 catalog_manager.cc:4841] TS a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907): tablet df4548b9bcaf451585de4f5202924442 (table test-table [id=e44da20ad3ac478d9e68830b5ef809c5]) successfully deleted
W20260926 09:35:04.394389 14102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38249 (error 108)
I20260926 09:35:04.394665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 14314
I20260926 09:35:04.395268 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.394776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.0.1:60454)
0926 09:35:04.395208 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.395243 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403}
W20260926 09:35:04.395876 14102 catalog_manager.cc:4584] TS c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df4548b9bcaf451585de4f5202924442: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38249: connect: Connection refused (error 111)
I20260926 09:35:04.399814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 14448
I20260926 09:35:04.404528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 14086
I20260926 09:35:04.409788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38091
--webserver_interface=127.0.98.62
--webserver_port=35907
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:04.491961 14621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:04.492172 14621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:04.492195 14621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:04.493305 14621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:35:04.493342 14621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:04.493356 14621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:04.493369 14621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:35:04.493381 14621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:35:04.494890 14621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:04.495091 14621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:04.495271 14621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:04.504007 14627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.504240 14621 server_base.cc:1034] running on GCE node
W20260926 09:35:04.504007 14630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:04.504004 14628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.504616 14621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:04.504941 14621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:04.506088 14621 hybrid_clock.cc:648] HybridClock initialized: now 1790415304506080 us; error 30 us; skew 500 ppm
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:04.508661 14621 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:35:04.509065 14621 webserver.cc:460] Webserver started at http://127.0.98.62:35907/ using document root <none> and password file <none>
I20260926 09:35:04.509271 14621 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:04.509332 14621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:04.510682 14621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:04.511368 14637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.511600 14621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:04.511699 14621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "30ea703f6d33468ab474eab36409a94c"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "cdc4797482a19fd23fe00c6aebe379cc"
server_key_iv: "235c4a55c82405151af3ebbcf4396ad7"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.512109 14621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:04.537464 14621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:04.539690 14621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:04.539849 14621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:04.544070 14621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38091
I20260926 09:35:04.544123 14689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38091 every 8 connection(s)
I20260926 09:35:04.544523 14621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:35:04.546293 14690 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:35:04.546903 14690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Bootstrap starting.
I20260926 09:35:04.554764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 14621
I20260926 09:35:04.555128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38249
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38091
--webserver_port=36963
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:35:04.555542 14690 log.cc:824] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:04.557717 14690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:04.558029 14690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Bootstrap complete.
I20260926 09:35:04.560572 14690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:04.560813 14690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ea703f6d33468ab474eab36409a94c, State: Initialized, Role: FOLLOWER
I20260926 09:35:04.560981 14690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:04.561060 14690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:35:04.561115 14690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:35:04.561165 14690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:35:04.562213 14690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:04.562305 14690 leader_election.cc:302] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ea703f6d33468ab474eab36409a94c; no voters:
I20260926 09:35:04.562538 14690 leader_election.cc:288] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:35:04.562573 14694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:35:04.562688 14694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 2 LEADER]: Becoming Leader. State: Replica: 30ea703f6d33468ab474eab36409a94c, State: Running, Role: LEADER
I20260926 09:35:04.562821 14690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:35:04.562845 14694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:04.563254 14696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "30ea703f6d33468ab474eab36409a94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } } }
I20260926 09:35:04.563378 14696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:04.563760 14695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ea703f6d33468ab474eab36409a94c. Latest consensus state: current_term: 2 leader_uuid: "30ea703f6d33468ab474eab36409a94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } } }
I20260926 09:35:04.563838 14695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:04.563987 14701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:35:04.564807 14701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:35:04.564954 14701 catalog_manager.cc:1259] Loaded cluster ID: 0029964988e94ef7bf0679e8314d30f4
I20260926 09:35:04.565223 14701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:35:04.565508 14701 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 09:35:04.570197 14701 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 09:35:04.572943 14701 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:35:04.573172 14701 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Loaded TSK: 0
I20260926 09:35:04.573418 14701 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:35:04.641793 14693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:04.642030 14693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:04.642069 14693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:04.643224 14693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:04.643290 14693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:04.643371 14693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:35:04.645061 14693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36963
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:04.645361 14693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:04.645598 14693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:04.651585 14717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.651743 14693 server_base.cc:1034] running on GCE node
W20260926 09:35:04.651638 14718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:04.651589 14720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.652241 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:04.652542 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:04.653726 14693 hybrid_clock.cc:648] HybridClock initialized: now 1790415304653700 us; error 39 us; skew 500 ppm
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:04.656847 14693 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:35:04.657393 14693 webserver.cc:460] Webserver started at http://127.0.98.1:36963/ using document root <none> and password file <none>
I20260926 09:35:04.657691 14693 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:04.657773 14693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:04.659565 14693 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:35:04.660388 14727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.660594 14693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:04.660702 14693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "c273af5c39254444ae0f2080092c18ed"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "a5c4a6b53addf91e4433d6f49ec9a3cd"
server_key_iv: "fe7743754c089f5c4dea952fa27cb6b9"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.661166 14693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:04.677983 14693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:04.680449 14693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:04.680610 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:04.680869 14693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:04.681373 14734 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:35:04.682912 14693 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:35:04.682977 14693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:35:04.683012 14693 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:35:04.683919 14693 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:35:04.683962 14693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:35:04.684044 14734 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Bootstrap starting.
I20260926 09:35:04.690419 14693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38249
I20260926 09:35:04.690610 14841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38249 every 8 connection(s)
I20260926 09:35:04.690868 14693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:35:04.695991 14734 log.cc:824] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:04.698724 14734 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:04.699083 14734 tablet_bootstrap.cc:492] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Bootstrap complete.
I20260926 09:35:04.700332 14734 ts_tablet_manager.cc:1397] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.014s
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:04.700839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 14693
I20260926 09:35:04.701128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44155
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38091
--webserver_port=34121
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:35:04.702001 14734 raft_consensus.cc:348] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.702510 14734 raft_consensus.cc:729] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c273af5c39254444ae0f2080092c18ed, State: Initialized, Role: FOLLOWER
I20260926 09:35:04.702750 14734 consensus_queue.cc:260] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.703130 14734 ts_tablet_manager.cc:1428] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:35:04.705256 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.691769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35965 (local address 127.0.98.62:38091)
0926 09:35:04.691964 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.691974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.692017 (+ 43us) server_negotiation.cc:407] Connection header received
0926 09:35:04.692762 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.692770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.692837 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.693008 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:04.693762 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.695567 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.697107 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.697785 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.697907 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.700981 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:04.700985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:04.700986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:04.700998 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:04.703139 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.703665 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.703668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.703668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.703777 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.704061 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.704062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.704062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.704307 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:04.704402 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.704793 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.704960 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260926 09:35:04.706115 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.691616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:35965)
0926 09:35:04.691849 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.691897 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.692496 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.693045 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.693050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.693384 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:04.693627 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.693633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.695704 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.695706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.696999 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.697003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.698022 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.698024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.698189 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.698857 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:04.698867 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:04.700851 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:04.703255 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.703260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.703262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.703554 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.703856 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.703857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.703857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.703972 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.704394 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:04.704401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:04.704579 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.705581 (+ 1002us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.705790 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":131,"threads_started":1}
I20260926 09:35:04.707063 14842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38091
I20260926 09:35:04.707165 14842 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:04.707403 14842 heartbeater.cc:511] Master 127.0.98.62:38091 requested a full tablet report, sending...
I20260926 09:35:04.708222 14654 ts_manager.cc:194] Registered new tserver with Master: c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249)
I20260926 09:35:04.709375 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35965
I20260926 09:35:04.710497 14842 heartbeater.cc:503] Master 127.0.98.62:38091 was elected leader, sending a full tablet report...
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:35:04.717115 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.708938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.0.98.1:38249)
0926 09:35:04.709109 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.709113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.709329 (+ 216us) server_negotiation.cc:407] Connection header received
0926 09:35:04.709411 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.709414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.709432 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.709515 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:04.710069 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.710852 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.711719 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.712006 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.712125 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.714068 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:04.714076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:04.714077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:04.714083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:04.715804 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.716233 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.716237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.716239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.716298 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.716503 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.716504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.716505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.716656 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:04.716749 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.716927 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.717018 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260926 09:35:04.717115 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.708877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.0.1:60464)
0926 09:35:04.709196 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.709235 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.709323 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.709533 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.709537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.709927 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:04.709984 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.709986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.710932 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.710933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.711618 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.711621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.712126 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.712130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.712211 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.712714 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:04.712718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:04.713978 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:04.715902 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.715908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.715910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.716111 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.716368 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.716369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.716369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.716437 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.716728 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:04.716734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:04.716785 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.716891 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.717015 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":84,"threads_started":1}
I20260926 09:35:04.717516 14776 tablet_service.cc:1514] Processing DeleteTablet for tablet df4548b9bcaf451585de4f5202924442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60464
I20260926 09:35:04.718027 14851 tablet_replica.cc:330] stopping tablet replica
I20260926 09:35:04.718149 14851 raft_consensus.cc:2229] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:04.718220 14851 raft_consensus.cc:2258] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:04.718525 14851 ts_tablet_manager.cc:1905] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:04.720588 14851 ts_tablet_manager.cc:1918] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:04.720665 14851 log.cc:1196] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/df4548b9bcaf451585de4f5202924442
I20260926 09:35:04.720902 14851 ts_tablet_manager.cc:1939] T df4548b9bcaf451585de4f5202924442 P c273af5c39254444ae0f2080092c18ed: Deleting consensus metadata
I20260926 09:35:04.721436 14638 catalog_manager.cc:4841] TS c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249): tablet df4548b9bcaf451585de4f5202924442 (table test-table [id=e44da20ad3ac478d9e68830b5ef809c5]) successfully deleted
W20260926 09:35:04.789912 14848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:04.790148 14848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:04.790189 14848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:04.791371 14848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:04.791435 14848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:04.791477 14848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:35:04.792955 14848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34121
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:04.793226 14848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:04.793457 14848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:04.798448 14856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:04.798456 14857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.798710 14848 server_base.cc:1034] running on GCE node
W20260926 09:35:04.798456 14859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.798991 14848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:04.799254 14848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:04.800453 14848 hybrid_clock.cc:648] HybridClock initialized: now 1790415304800446 us; error 31 us; skew 500 ppm
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:04.802860 14848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:35:04.803299 14848 webserver.cc:460] Webserver started at http://127.0.98.2:34121/ using document root <none> and password file <none>
I20260926 09:35:04.803578 14848 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:04.803663 14848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:04.804988 14848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:04.805569 14866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.805734 14848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:35:04.805831 14848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "994dd2b9231e4ce3b432c340226989a3"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "2bd9eea4cbd7fe1044a7de23d87e69c8"
server_key_iv: "3e86af2e285ab8a5b57adb9a9eaa537a"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.806181 14848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:04.817910 14848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:04.819813 14848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:04.819981 14848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:04.820209 14848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:04.820535 14848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:04.820590 14848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.820631 14848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:04.820659 14848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.826583 14848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44155
I20260926 09:35:04.826634 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44155 every 8 connection(s)
I20260926 09:35:04.826963 14848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:35:04.828011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 14848
I20260926 09:35:04.828289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37907
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38091
--webserver_port=40147
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:04.836614 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.827706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50207 (local address 127.0.98.62:38091)
0926 09:35:04.827761 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.827764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.827981 (+ 217us) server_negotiation.cc:407] Connection header received
0926 09:35:04.828653 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.828654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.828667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.828721 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:04.829665 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.830176 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.831179 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.831506 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.831594 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.833708 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:04.833710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:04.833710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:04.833719 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:04.835161 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.835631 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.835632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.835633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.835695 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.835956 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.835957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.835958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.836075 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:04.836153 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.836430 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.836520 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:35:04.836794 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.827699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:50207)
0926 09:35:04.827865 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.827899 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.828549 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.828796 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.828800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.829372 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:04.829562 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.829566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.830278 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.830279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.831085 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.831087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.831605 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.831606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.831741 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.832282 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:04.832290 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:04.833609 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:04.835265 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.835278 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.835281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.835552 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.835765 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.835766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.835766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.835880 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.836143 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:04.836149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:04.836298 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.836410 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.836562 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260926 09:35:04.837284 14980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38091
I20260926 09:35:04.837396 14980 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:04.837582 14980 heartbeater.cc:511] Master 127.0.98.62:38091 requested a full tablet report, sending...
I20260926 09:35:04.837971 14654 ts_manager.cc:194] Registered new tserver with Master: 994dd2b9231e4ce3b432c340226989a3 (127.0.98.2:44155)
I20260926 09:35:04.838465 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50207
W20260926 09:35:04.909464 14983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:04.909662 14983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:04.909684 14983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:04.910854 14983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:04.910897 14983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:04.910924 14983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:35:04.912356 14983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40147
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:04.912587 14983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:04.912765 14983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:04.917732 14989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:04.917814 14991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:04.917783 14988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:04.917964 14983 server_base.cc:1034] running on GCE node
I20260926 09:35:04.918193 14983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:04.918462 14983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:04.919633 14983 hybrid_clock.cc:648] HybridClock initialized: now 1790415304919631 us; error 38 us; skew 500 ppm
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:04.922101 14983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:35:04.922529 14983 webserver.cc:460] Webserver started at http://127.0.98.3:40147/ using document root <none> and password file <none>
I20260926 09:35:04.922752 14983 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:04.922839 14983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:04.924202 14983 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:04.924789 14998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.924957 14983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:35:04.925060 14983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "a68137e848ee4643b02ee70de19c9dcc"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "57acbd6fd7eb3fb1f5611a734f24437f"
server_key_iv: "610c8f1c0bf9f11e689da7a8d4645fe4"
server_key_version: "encryptionkey@0"
I20260926 09:35:04.925412 14983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:04.940246 14983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:04.942502 14983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:04.942608 14983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:04.942775 14983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:04.943073 14983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:04.943130 14983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.943172 14983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:04.943197 14983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:04.948730 14983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37907
I20260926 09:35:04.948827 15111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37907 every 8 connection(s)
I20260926 09:35:04.949152 14983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:35:04.953065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 14983
Sep 26 09:35:04 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:04.958925 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.949822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50103 (local address 127.0.98.62:38091)
0926 09:35:04.949877 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.949878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.950146 (+ 268us) server_negotiation.cc:407] Connection header received
0926 09:35:04.950686 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.950688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.950702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.950753 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:04.951789 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.952271 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.953319 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.953640 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.953723 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.956050 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:04.956051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:04.956052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:04.956059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:04.957592 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.957969 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.957971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.957971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.958031 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.958288 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.958289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.958289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.958402 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:04.958472 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.958744 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.958823 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:35:04.959105 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.949833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:50103)
0926 09:35:04.949992 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.950053 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.950568 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.950823 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.950827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.951489 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:04.951679 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.951684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.952373 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.952374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.953216 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.953219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.953729 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.953730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.953879 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.954413 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:04.954421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:04.955966 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:04.957668 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.957675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.957677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.957892 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.958100 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:04.958100 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:04.958101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:04.958204 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:04.958479 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:04.958485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:04.958610 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.958716 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.958869 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260926 09:35:04.959625 15112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38091
I20260926 09:35:04.959731 15112 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:04.959929 15112 heartbeater.cc:511] Master 127.0.98.62:38091 requested a full tablet report, sending...
I20260926 09:35:04.960144 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.953485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.0.98.62:38091)
0926 09:35:04.953582 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.953584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.953624 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:35:04.953715 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.953716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.953728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.953817 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:04.954310 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.954782 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.955627 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.955916 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.955988 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.957363 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:04.957364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:04.957364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:04.957370 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:04.958732 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.959283 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.959285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.959285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.959388 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:04.959622 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:04.959623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:04.959623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:04.959762 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:04.959922 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.959962 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.960040 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":55,"threads_started":1}
I20260926 09:35:04.960479 14654 ts_manager.cc:194] Registered new tserver with Master: a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907)
I20260926 09:35:04.960850 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50103
I20260926 09:35:04.960934 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:35:04.964046 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.961318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.0.98.62:38091)
0926 09:35:04.961355 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.961357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.961474 (+ 117us) server_negotiation.cc:407] Connection header received
0926 09:35:04.961524 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.961525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.961559 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.961637 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:35:04.961713 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.962213 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.963190 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.963510 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.963577 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.963736 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:35:04.963837 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:35:04.963902 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.963959 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.963981 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:35:04.964396 14654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:35:04.966336 14654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:35:04.968590 14776 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc30c179b545b8870a8ba623793cdd (DEFAULT_TABLE table=test-table [id=e328e03c6de64e9fad9fba4e7964c9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:04.968820 14776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc30c179b545b8870a8ba623793cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:04.970322 14734 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Bootstrap starting.
I20260926 09:35:04.970647 14734 tablet_bootstrap.cc:654] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:04.971230 14734 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: No bootstrap required, opened a new log
I20260926 09:35:04.971292 14734 ts_tablet_manager.cc:1397] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:04.971704 14734 raft_consensus.cc:348] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.971781 14734 raft_consensus.cc:374] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:04.971830 14734 raft_consensus.cc:729] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c273af5c39254444ae0f2080092c18ed, State: Initialized, Role: FOLLOWER
I20260926 09:35:04.971890 14734 consensus_queue.cc:260] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.972002 14734 ts_tablet_manager.cc:1428] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:04.972254 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.968136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37907 (local address 127.0.0.1:58538)
0926 09:35:04.968234 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.968245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.968302 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.968651 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.968652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.968654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:04.968701 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.968703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.970051 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.970052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.970910 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.970913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.971958 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.971959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.972030 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.972052 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.972118 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.972165 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 09:35:04.972277 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.968224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.0.98.3:37907)
0926 09:35:04.968478 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.968482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.968493 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:35:04.968536 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.968538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.968554 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.968632 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:04.968785 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.969915 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.971011 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.971844 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.971955 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.972027 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.972134 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.972180 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":68,"threads_started":1}
I20260926 09:35:04.972429 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.968137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44155 (local address 127.0.0.1:55598)
0926 09:35:04.968306 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:04.968316 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:04.968356 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:04.968748 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:04.968749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:04.968751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:04.968795 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.968797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.970049 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.970050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.971180 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:04.971183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.972106 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.972107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:04.972159 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.972162 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:35:04.972189 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 09:35:04.972218 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":45,"thread_start_us":99,"threads_started":1}
I20260926 09:35:04.972546 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:04.968404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.0.98.2:44155)
0926 09:35:04.968593 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:04.968598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:04.968608 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:35:04.968647 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:04.968649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:04.968664 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:04.968747 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:04.968869 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.969918 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.971268 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:04.972008 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:04.972299 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:04.972374 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:04.972403 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 09:35:04.972446 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260926 09:35:04.972981 15046 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc30c179b545b8870a8ba623793cdd (DEFAULT_TABLE table=test-table [id=e328e03c6de64e9fad9fba4e7964c9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:04.973363 15046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc30c179b545b8870a8ba623793cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:04.973363 14914 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc30c179b545b8870a8ba623793cdd (DEFAULT_TABLE table=test-table [id=e328e03c6de64e9fad9fba4e7964c9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:04.973706 14914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc30c179b545b8870a8ba623793cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:04.976181 15122 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Bootstrap starting.
I20260926 09:35:04.976336 15123 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Bootstrap starting.
I20260926 09:35:04.977051 15123 tablet_bootstrap.cc:654] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:04.977054 15122 tablet_bootstrap.cc:654] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:04.977530 15123 log.cc:824] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:04.977599 15122 log.cc:824] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:04.978264 15123 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: No bootstrap required, opened a new log
I20260926 09:35:04.978317 15123 ts_tablet_manager.cc:1397] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:35:04.978336 15122 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: No bootstrap required, opened a new log
I20260926 09:35:04.978407 15122 ts_tablet_manager.cc:1397] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:04.979699 15123 raft_consensus.cc:348] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.979858 15123 raft_consensus.cc:374] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:04.979773 15122 raft_consensus.cc:348] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.979907 15123 raft_consensus.cc:729] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994dd2b9231e4ce3b432c340226989a3, State: Initialized, Role: FOLLOWER
I20260926 09:35:04.979935 15122 raft_consensus.cc:374] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:04.979984 15122 raft_consensus.cc:729] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68137e848ee4643b02ee70de19c9dcc, State: Initialized, Role: FOLLOWER
I20260926 09:35:04.980052 15123 consensus_queue.cc:260] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.980113 15122 consensus_queue.cc:260] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:04.980334 15123 ts_tablet_manager.cc:1428] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:04.980438 15112 heartbeater.cc:503] Master 127.0.98.62:38091 was elected leader, sending a full tablet report...
I20260926 09:35:04.980460 14980 heartbeater.cc:503] Master 127.0.98.62:38091 was elected leader, sending a full tablet report...
I20260926 09:35:04.980674 15122 ts_tablet_manager.cc:1428] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260926 09:35:05.077509 14981 tablet.cc:2369] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:35:05.079046 15126 raft_consensus.cc:482] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:35:05.079177 15126 raft_consensus.cc:504] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:05.079599 15126 leader_election.cc:288] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907), c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249)
I20260926 09:35:05.083338 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.079928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51979 (local address 127.0.98.3:37907)
0926 09:35:05.079982 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:05.079984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:05.079994 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:35:05.080026 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:05.080028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:05.080039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:05.080102 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:05.080280 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.081054 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.082414 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.082947 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.083025 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:05.083036 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:05.083200 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:35:05.083246 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:35:05.083354 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.079897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39057 (local address 127.0.98.1:38249)
0926 09:35:05.079952 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:05.079954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:05.080181 (+ 227us) server_negotiation.cc:407] Connection header received
0926 09:35:05.080222 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:05.080223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:05.080263 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:05.080339 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:05.080477 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.081194 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.082414 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.083032 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.083098 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:05.083174 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:05.083230 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 09:35:05.083265 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:35:05.083370 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.079819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37907 (local address 127.0.98.2:51979)
0926 09:35:05.079874 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.079887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:05.079939 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:05.080128 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:05.080129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:05.080131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:05.080207 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:05.080209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.081134 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.081134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:05.082318 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:05.082322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.083029 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.083029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:05.083120 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:05.083124 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:35:05.083181 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:35:05.083229 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:35:05.083377 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.079839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.98.2:39057)
0926 09:35:05.080104 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.080114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:05.080167 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:05.080361 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:05.080362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:05.080363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:05.080412 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:05.080413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.081258 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.081259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:05.082341 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:05.082343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.083106 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.083107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:05.083169 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:05.083173 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:35:05.083216 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 09:35:05.083243 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":90,"thread_start_us":102,"threads_started":1}
I20260926 09:35:05.083782 15066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc30c179b545b8870a8ba623793cdd" candidate_uuid: "994dd2b9231e4ce3b432c340226989a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68137e848ee4643b02ee70de19c9dcc" is_pre_election: true
I20260926 09:35:05.083889 14796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc30c179b545b8870a8ba623793cdd" candidate_uuid: "994dd2b9231e4ce3b432c340226989a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c273af5c39254444ae0f2080092c18ed" is_pre_election: true
I20260926 09:35:05.083971 15066 raft_consensus.cc:2454] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 994dd2b9231e4ce3b432c340226989a3 in term 0.
I20260926 09:35:05.084018 14796 raft_consensus.cc:2454] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 994dd2b9231e4ce3b432c340226989a3 in term 0.
I20260926 09:35:05.084232 14870 leader_election.cc:302] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994dd2b9231e4ce3b432c340226989a3, a68137e848ee4643b02ee70de19c9dcc; no voters:
I20260926 09:35:05.084398 15126 raft_consensus.cc:2781] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:35:05.084460 15126 raft_consensus.cc:482] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:35:05.084493 15126 raft_consensus.cc:3037] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:05.085242 15126 raft_consensus.cc:504] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:05.085379 15126 leader_election.cc:288] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 1 election: Requested vote from peers a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907), c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249)
I20260926 09:35:05.085500 15066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc30c179b545b8870a8ba623793cdd" candidate_uuid: "994dd2b9231e4ce3b432c340226989a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68137e848ee4643b02ee70de19c9dcc"
I20260926 09:35:05.085530 14796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc30c179b545b8870a8ba623793cdd" candidate_uuid: "994dd2b9231e4ce3b432c340226989a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c273af5c39254444ae0f2080092c18ed"
I20260926 09:35:05.085587 15066 raft_consensus.cc:3037] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:05.085613 14796 raft_consensus.cc:3037] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:05.086295 14796 raft_consensus.cc:2454] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994dd2b9231e4ce3b432c340226989a3 in term 1.
I20260926 09:35:05.086300 15066 raft_consensus.cc:2454] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994dd2b9231e4ce3b432c340226989a3 in term 1.
I20260926 09:35:05.086484 14867 leader_election.cc:302] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994dd2b9231e4ce3b432c340226989a3, c273af5c39254444ae0f2080092c18ed; no voters:
I20260926 09:35:05.086570 15126 raft_consensus.cc:2781] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:35:05.086706 15126 raft_consensus.cc:686] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 LEADER]: Becoming Leader. State: Replica: 994dd2b9231e4ce3b432c340226989a3, State: Running, Role: LEADER
I20260926 09:35:05.086807 15126 consensus_queue.cc:237] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:05.087625 14653 catalog_manager.cc:5495] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 994dd2b9231e4ce3b432c340226989a3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "994dd2b9231e4ce3b432c340226989a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } health_report { overall_health: UNKNOWN } } }
I20260926 09:35:05.118216 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.116076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.0.98.2:44155)
0926 09:35:05.116133 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:05.116135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:05.116144 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:35:05.116190 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:05.116191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:05.116211 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:05.116276 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:35:05.116411 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.116880 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.117570 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:05.117754 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:05.117807 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:05.117938 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:35:05.118039 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:35:05.118101 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:05.118135 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 09:35:05.118153 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:35:05.120107 14796 raft_consensus.cc:1261] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Refusing update from remote peer 994dd2b9231e4ce3b432c340226989a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:35:05.120110 15066 raft_consensus.cc:1261] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 1 FOLLOWER]: Refusing update from remote peer 994dd2b9231e4ce3b432c340226989a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:35:05.120486 15126 consensus_queue.cc:1035] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:35:05.120646 15129 consensus_queue.cc:1035] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:35:05.123005 15132 mvcc.cc:205] Tried to move back new op lower bound from 7333541089769156608 to 7333541089636270080. Current Snapshot: MvccSnapshot[applied={T|T < 7333541089769156608}]
I20260926 09:35:05.125149 14653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:35:05.125257 14653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:35:05.125963 14653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Sending DeleteTablet for 3 replicas of tablet 31cc30c179b545b8870a8ba623793cdd
I20260926 09:35:05.126266 15046 tablet_service.cc:1514] Processing DeleteTablet for tablet 31cc30c179b545b8870a8ba623793cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58538
I20260926 09:35:05.126271 14914 tablet_service.cc:1514] Processing DeleteTablet for tablet 31cc30c179b545b8870a8ba623793cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55598
I20260926 09:35:05.126353 14776 tablet_service.cc:1514] Processing DeleteTablet for tablet 31cc30c179b545b8870a8ba623793cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60464
I20260926 09:35:05.126456 14851 tablet_replica.cc:330] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: stopping tablet replica
I20260926 09:35:05.126518 14851 raft_consensus.cc:2229] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:05.126509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 14693
I20260926 09:35:05.126573 14851 raft_consensus.cc:2258] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:05.127192 15141 tablet_replica.cc:330] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: stopping tablet replica
I20260926 09:35:05.127298 15141 raft_consensus.cc:2229] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:05.127424 15141 raft_consensus.cc:2258] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:05.127866 15141 ts_tablet_manager.cc:1905] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:05.129904 15141 ts_tablet_manager.cc:1918] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:05.129997 15141 log.cc:1196] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/31cc30c179b545b8870a8ba623793cdd
I20260926 09:35:05.130313 15141 ts_tablet_manager.cc:1939] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Deleting consensus metadata
I20260926 09:35:05.130745 14641 catalog_manager.cc:4841] TS a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907): tablet 31cc30c179b545b8870a8ba623793cdd (table test-table [id=e328e03c6de64e9fad9fba4e7964c9da]) successfully deleted
W20260926 09:35:05.133430 14638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38249 (error 108)
I20260926 09:35:05.133711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 14848
I20260926 09:35:05.134294 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.133797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.0.1:60468)
0926 09:35:05.134202 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.134243 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
W20260926 09:35:05.135425 14638 catalog_manager.cc:4584] TS c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31cc30c179b545b8870a8ba623793cdd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38249: connect: Connection refused (error 111)
I20260926 09:35:05.138651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 14983
I20260926 09:35:05.138728 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.138600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44155 (local address 127.0.0.1:55616)
0926 09:35:05.138653 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.138681 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:35:05.143179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 14621
I20260926 09:35:05.148499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38091
--webserver_interface=127.0.98.62
--webserver_port=35907
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:05.229960 15144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:05.230228 15144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:05.230268 15144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:05.231540 15144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:35:05.231616 15144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:05.231642 15144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:05.231664 15144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:35:05.231693 15144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:35:05.233191 15144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:05.233481 15144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:05.233716 15144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:05.238592 15150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:05.238674 15152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:05.238757 15144 server_base.cc:1034] running on GCE node
W20260926 09:35:05.238590 15149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:05.239123 15144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:05.239480 15144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:05.240631 15144 hybrid_clock.cc:648] HybridClock initialized: now 1790415305240621 us; error 32 us; skew 500 ppm
Sep 26 09:35:05 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:05.243222 15144 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:35:05.243685 15144 webserver.cc:460] Webserver started at http://127.0.98.62:35907/ using document root <none> and password file <none>
I20260926 09:35:05.243904 15144 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:05.243947 15144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:05.245211 15144 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:05.245914 15159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:05.246093 15144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:05.246200 15144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "30ea703f6d33468ab474eab36409a94c"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "cdc4797482a19fd23fe00c6aebe379cc"
server_key_iv: "235c4a55c82405151af3ebbcf4396ad7"
server_key_version: "encryptionkey@0"
I20260926 09:35:05.246598 15144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:05.266636 15144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:05.269131 15144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:05.269307 15144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:05.273526 15144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38091
I20260926 09:35:05.273563 15211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38091 every 8 connection(s)
I20260926 09:35:05.273975 15144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:35:05.275621 15212 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:35:05.276315 15212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Bootstrap starting.
I20260926 09:35:05.283568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 15144
I20260926 09:35:05.283957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38249
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38091
--webserver_port=36963
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:35:05.285108 15212 log.cc:824] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:05.287834 15212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:05.288172 15212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c: Bootstrap complete.
I20260926 09:35:05.290484 15212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:05.290714 15212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ea703f6d33468ab474eab36409a94c, State: Initialized, Role: FOLLOWER
I20260926 09:35:05.290889 15212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:05.290962 15212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:35:05.291028 15212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:35:05.291074 15212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 2 FOLLOWER]: Advancing to term 3
I20260926 09:35:05.292280 15212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:05.292379 15212 leader_election.cc:302] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ea703f6d33468ab474eab36409a94c; no voters:
I20260926 09:35:05.292595 15212 leader_election.cc:288] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 09:35:05.292634 15216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 3 FOLLOWER]: Leader election won for term 3
I20260926 09:35:05.292747 15216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [term 3 LEADER]: Becoming Leader. State: Replica: 30ea703f6d33468ab474eab36409a94c, State: Running, Role: LEADER
I20260926 09:35:05.292843 15216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260926 09:35:05.292881 15212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:35:05.293109 15216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ea703f6d33468ab474eab36409a94c. Latest consensus state: current_term: 3 leader_uuid: "30ea703f6d33468ab474eab36409a94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } } }
I20260926 09:35:05.293186 15216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:05.293118 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "30ea703f6d33468ab474eab36409a94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ea703f6d33468ab474eab36409a94c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } } }
I20260926 09:35:05.293242 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ea703f6d33468ab474eab36409a94c [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:05.293500 15220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:35:05.294612 15220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:35:05.294898 15220 catalog_manager.cc:1259] Loaded cluster ID: 0029964988e94ef7bf0679e8314d30f4
I20260926 09:35:05.294978 15220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 09:35:05.295185 15220 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1609a6a9ad google::LogMessage::Fail() at ??:0
@ 0x7f1609a6d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1609a6a4de google::LogMessage::Flush() at ??:0
@ 0x7f1609a6ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f161198b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f16119948b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f160b9380d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f160b92d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f160c7916db start_thread at ??:0
@ 0x7f160856861f clone at ??:0
W20260926 09:35:05.387712 15214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:05.387894 15214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:05.387917 15214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:05.389048 15214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:05.389091 15214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:05.389158 15214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:35:05.390628 15214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36963
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:05.390858 15214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:05.391109 15214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:05.396606 15251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:05.396632 15248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:05.396803 15214 server_base.cc:1034] running on GCE node
W20260926 09:35:05.396816 15249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:05.397073 15214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:05.397353 15214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:05.398522 15214 hybrid_clock.cc:648] HybridClock initialized: now 1790415305398498 us; error 40 us; skew 500 ppm
Sep 26 09:35:05 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:05.401019 15214 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:35:05.401487 15214 webserver.cc:460] Webserver started at http://127.0.98.1:36963/ using document root <none> and password file <none>
I20260926 09:35:05.401734 15214 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:05.401809 15214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:05.403167 15214 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:05.403946 15258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:05.404166 15214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:35:05.404260 15214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "c273af5c39254444ae0f2080092c18ed"
format_stamp: "Formatted at 2026-09-26 09:35:03 on dist-test-slave-21h8"
server_key: "a5c4a6b53addf91e4433d6f49ec9a3cd"
server_key_iv: "fe7743754c089f5c4dea952fa27cb6b9"
server_key_version: "encryptionkey@0"
I20260926 09:35:05.404738 15214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:05.427001 15214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:05.429492 15214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:05.429649 15214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:05.429905 15214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:05.430390 15265 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:35:05.431923 15214 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:35:05.431983 15214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:35:05.432018 15214 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:35:05.432870 15214 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:35:05.432916 15214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:35:05.432992 15265 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Bootstrap starting.
I20260926 09:35:05.440153 15214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38249
I20260926 09:35:05.440197 15372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38249 every 8 connection(s)
I20260926 09:35:05.440646 15214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:35:05.443647 15265 log.cc:824] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:05.445648 15265 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:05.445920 15265 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Bootstrap complete.
I20260926 09:35:05.446822 15265 ts_tablet_manager.cc:1397] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260926 09:35:05.447937 15265 raft_consensus.cc:348] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:05.448182 15265 raft_consensus.cc:729] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c273af5c39254444ae0f2080092c18ed, State: Initialized, Role: FOLLOWER
I20260926 09:35:05.448366 15265 consensus_queue.cc:260] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:05.448661 15265 ts_tablet_manager.cc:1428] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260926 09:35:05.450109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 15214
I20260926 09:35:05.450383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44155
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38091
--webserver_port=34121
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:05.505635 15375 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:35:05.441425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:50523)
0926 09:35:05.441678 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.441732 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:05.442173 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:05.505429 (+ 63256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":129,"threads_started":1}
W20260926 09:35:05.505906 15373 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38091 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38091: Client connection negotiation failed: client connection to 127.0.98.62:38091: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:35:05.506292 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.506173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:57775)
0926 09:35:05.506228 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.506272 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:35:05.506654 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.506570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:43249)
0926 09:35:05.506616 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.506639 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:35:05.506788 15373 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:35:05.537077 15378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:05.537304 15378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:05.537329 15378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:05.538563 15378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:05.538616 15378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:05.538674 15378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:35:05.540220 15378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34121
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:05.540625 15378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:05.540902 15378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:05.546016 15384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:05.546056 15378 server_base.cc:1034] running on GCE node
W20260926 09:35:05.546028 15383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:05.546082 15386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:05.546417 15378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:05.546634 15378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:05.547844 15378 hybrid_clock.cc:648] HybridClock initialized: now 1790415305547829 us; error 30 us; skew 500 ppm
Sep 26 09:35:05 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:05.550429 15378 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:35:05.550869 15378 webserver.cc:460] Webserver started at http://127.0.98.2:34121/ using document root <none> and password file <none>
I20260926 09:35:05.551115 15378 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:05.551160 15378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:05.552584 15378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:05.553274 15393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:05.553444 15378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:35:05.553551 15378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "994dd2b9231e4ce3b432c340226989a3"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "2bd9eea4cbd7fe1044a7de23d87e69c8"
server_key_iv: "3e86af2e285ab8a5b57adb9a9eaa537a"
server_key_version: "encryptionkey@0"
I20260926 09:35:05.553907 15378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:05.566587 15378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:05.568490 15378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:05.568655 15378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:05.568890 15378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:05.569301 15400 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:35:05.570770 15378 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:35:05.570832 15378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:35:05.570878 15378 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:35:05.571559 15378 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:35:05.571619 15378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:05.571674 15400 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Bootstrap starting.
I20260926 09:35:05.578917 15378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44155
I20260926 09:35:05.579031 15507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44155 every 8 connection(s)
I20260926 09:35:05.579397 15378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:35:05.580569 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.580042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:46573)
0926 09:35:05.580243 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.580340 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
W20260926 09:35:05.580803 15508 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38091 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38091: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
I20260926 09:35:05.581198 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.581001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:44613)
0926 09:35:05.581130 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.581167 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260926 09:35:05.581310 15400 log.cc:824] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:05.581530 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.581454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:44537)
0926 09:35:05.581484 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.581509 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 09:35:05.581678 15508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:35:05.583498 15400 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:05.583777 15400 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Bootstrap complete.
I20260926 09:35:05.584694 15400 ts_tablet_manager.cc:1397] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260926 09:35:05.585563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 15378
I20260926 09:35:05.585812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37907
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38091
--webserver_port=40147
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:35:05.586089 15400 raft_consensus.cc:348] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:05.586376 15400 raft_consensus.cc:729] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994dd2b9231e4ce3b432c340226989a3, State: Initialized, Role: FOLLOWER
I20260926 09:35:05.586531 15400 consensus_queue.cc:260] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:05.586760 15400 ts_tablet_manager.cc:1428] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:35:05.587042 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.586922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:52467)
0926 09:35:05.586977 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.587013 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:35:05.587354 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.587240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:55059)
0926 09:35:05.587287 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.587331 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:35:05.670183 15514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:05.670418 15514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:05.670459 15514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:05.671943 15514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:05.672009 15514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:05.672055 15514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:35:05.673529 15514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40147
--tserver_master_addrs=127.0.98.62:38091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:05.673717 15514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:05.673894 15514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:05.678846 15520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:05.678859 15522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:05.678859 15519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:05.678953 15514 server_base.cc:1034] running on GCE node
I20260926 09:35:05.679292 15514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:05.679589 15514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:05.680742 15514 hybrid_clock.cc:648] HybridClock initialized: now 1790415305680733 us; error 39 us; skew 500 ppm
Sep 26 09:35:05 dist-test-slave-21h8 krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:05.683125 15514 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:35:05.683599 15514 webserver.cc:460] Webserver started at http://127.0.98.3:40147/ using document root <none> and password file <none>
I20260926 09:35:05.683830 15514 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:05.683897 15514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:05.685137 15514 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:05.685702 15529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:05.685878 15514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:35:05.685976 15514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "a68137e848ee4643b02ee70de19c9dcc"
format_stamp: "Formatted at 2026-09-26 09:35:04 on dist-test-slave-21h8"
server_key: "57acbd6fd7eb3fb1f5611a734f24437f"
server_key_iv: "610c8f1c0bf9f11e689da7a8d4645fe4"
server_key_version: "encryptionkey@0"
I20260926 09:35:05.686300 15514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:05.707253 15514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:05.709595 15514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:05.709746 15514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:05.709964 15514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:05.710330 15514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:05.710381 15514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:05.710423 15514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:05.710451 15514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:05.716113 15514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37907
I20260926 09:35:05.716152 15642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37907 every 8 connection(s)
I20260926 09:35:05.716512 15514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:35:05.717563 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.717180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:46661)
0926 09:35:05.717347 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.717418 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
W20260926 09:35:05.717960 15643 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38091 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38091: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
I20260926 09:35:05.718248 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.718160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:58995)
0926 09:35:05.718206 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.718232 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:35:05.718500 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:05.718442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:45639)
0926 09:35:05.718461 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:05.718488 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260926 09:35:05.718624 15643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:35:05.720979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 15514
I20260926 09:35:06.507615 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:06.507388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:44217)
0926 09:35:06.507512 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:06.507553 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260926 09:35:06.587935 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:06.587699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:39927)
0926 09:35:06.587857 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:06.587894 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":59,"threads_started":1}
I20260926 09:35:06.719229 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:06.718966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:56553)
0926 09:35:06.719140 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:06.719180 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260926 09:35:06.881295 15650 raft_consensus.cc:482] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:35:06.881449 15650 raft_consensus.cc:504] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:06.881795 15650 leader_election.cc:288] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907), c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249)
Sep 26 09:35:06 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 09:35:06 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:35:06.893602 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:06.881915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.98.2:35493)
0926 09:35:06.882063 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:06.882072 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:06.882216 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:06.882563 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:06.882567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:06.882849 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:06.883101 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:06.883109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:06.884776 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:06.884777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:06.886052 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:06.886055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:06.886862 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:06.886863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:06.886995 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:06.887679 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:06.887683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:06.890007 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:06.892139 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:06.892139 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:06.892140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:06.892385 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:06.892668 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:06.892668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:06.892669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:06.892748 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:06.893149 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:06.893150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:06.893209 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 09:35:06.893351 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 09:35:06.893497 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":122,"thread_start_us":57,"threads_started":1}
I20260926 09:35:06.893618 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:06.882046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59691 (local address 127.0.98.3:37907)
0926 09:35:06.882199 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:06.882208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:06.882329 (+ 121us) server_negotiation.cc:407] Connection header received
0926 09:35:06.882841 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:06.882843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:06.882882 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:06.882985 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:06.883331 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:06.884669 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:06.886143 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:06.886763 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:06.886876 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:06.889593 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:06.889598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:06.889598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:06.889609 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:06.891733 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:06.892241 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:06.892244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:06.892244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:06.892352 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:06.892668 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:06.892669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:06.892669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:06.892993 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:06.893054 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:06.893384 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 09:35:06.893509 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260926 09:35:06.893664 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:06.882045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35493 (local address 127.0.98.1:38249)
0926 09:35:06.882195 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:06.882201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:06.882214 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:35:06.882397 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:06.882400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:06.882424 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:06.882513 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:06.883215 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:06.884669 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:06.886143 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:06.886763 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:06.886876 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:06.890153 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:06.890157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:06.890158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:06.890169 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:06.892028 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:06.892467 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:06.892470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:06.892470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:06.892566 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:06.892832 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:06.892832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:06.892833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:06.893075 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:06.893157 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:06.893419 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 09:35:06.893558 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
I20260926 09:35:06.893781 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:06.881914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37907 (local address 127.0.98.2:59691)
0926 09:35:06.881970 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:06.882001 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:06.882200 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:06.883003 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:06.883004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:06.883153 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:06.883198 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:06.883199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:06.884779 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:06.884780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:06.886053 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:06.886055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:06.886864 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:06.886865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:06.886995 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:06.887626 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:06.887653 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:06.889496 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:06.891819 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:06.891826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:06.891828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:06.892158 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:06.892454 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:06.892455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:06.892455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:06.892586 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:06.893065 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:06.893072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:06.893210 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 09:35:06.893300 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:35:06.893579 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":177}
I20260926 09:35:06.894795 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc30c179b545b8870a8ba623793cdd" candidate_uuid: "994dd2b9231e4ce3b432c340226989a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c273af5c39254444ae0f2080092c18ed" is_pre_election: true
I20260926 09:35:06.894979 15324 raft_consensus.cc:2454] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 994dd2b9231e4ce3b432c340226989a3 in term 1.
W20260926 09:35:06.895146 15397 leader_election.cc:341] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907): Not found: Tablet not found: 31cc30c179b545b8870a8ba623793cdd
I20260926 09:35:06.895222 15397 leader_election.cc:302] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 994dd2b9231e4ce3b432c340226989a3, c273af5c39254444ae0f2080092c18ed; no voters: a68137e848ee4643b02ee70de19c9dcc
I20260926 09:35:06.895358 15650 raft_consensus.cc:2781] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 09:35:06.895417 15650 raft_consensus.cc:482] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:35:06.895464 15650 raft_consensus.cc:3037] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:35:06.896550 15650 raft_consensus.cc:504] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:06.896716 15650 leader_election.cc:288] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 2 election: Requested vote from peers a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907), c273af5c39254444ae0f2080092c18ed (127.0.98.1:38249)
I20260926 09:35:06.896876 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc30c179b545b8870a8ba623793cdd" candidate_uuid: "994dd2b9231e4ce3b432c340226989a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c273af5c39254444ae0f2080092c18ed"
I20260926 09:35:06.896975 15324 raft_consensus.cc:3037] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 1 FOLLOWER]: Advancing to term 2
W20260926 09:35:06.897070 15397 leader_election.cc:341] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907): Not found: Tablet not found: 31cc30c179b545b8870a8ba623793cdd
I20260926 09:35:06.898049 15324 raft_consensus.cc:2454] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994dd2b9231e4ce3b432c340226989a3 in term 2.
I20260926 09:35:06.898205 15394 leader_election.cc:302] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 994dd2b9231e4ce3b432c340226989a3, c273af5c39254444ae0f2080092c18ed; no voters: a68137e848ee4643b02ee70de19c9dcc
I20260926 09:35:06.898309 15650 raft_consensus.cc:2781] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:35:06.898435 15650 raft_consensus.cc:686] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [term 2 LEADER]: Becoming Leader. State: Replica: 994dd2b9231e4ce3b432c340226989a3, State: Running, Role: LEADER
I20260926 09:35:06.898540 15650 consensus_queue.cc:237] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:06.898717 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:06.898621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:37069)
0926 09:35:06.898665 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:06.898689 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 09:35:07.321542 15324 raft_consensus.cc:1261] T 31cc30c179b545b8870a8ba623793cdd P c273af5c39254444ae0f2080092c18ed [term 2 FOLLOWER]: Refusing update from remote peer 994dd2b9231e4ce3b432c340226989a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260926 09:35:07.321573 15397 consensus_peers.cc:487] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 -> Peer a68137e848ee4643b02ee70de19c9dcc (127.0.98.3:37907): Couldn't send request to peer a68137e848ee4643b02ee70de19c9dcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31cc30c179b545b8870a8ba623793cdd. This is attempt 1: this message will repeat every 5th retry.
I20260926 09:35:07.321980 15654 consensus_queue.cc:1035] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 09:35:07.322996 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:07.322820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:37133)
0926 09:35:07.322898 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:07.322947 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63,"thread_start_us":48,"threads_started":1}
I20260926 09:35:07.323863 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:07.323738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:46685)
0926 09:35:07.323791 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:07.323828 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:35:07.324076 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:07.323980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:39243)
0926 09:35:07.324016 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:07.324050 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 09:35:07.703776 15662 ts_tablet_manager.cc:927] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Initiating tablet copy from peer 994dd2b9231e4ce3b432c340226989a3 (127.0.98.2:44155)
I20260926 09:35:07.704049 15662 tablet_copy_client.cc:316] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:44155
Sep 26 09:35:07 dist-test-slave-21h8 krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 09:35:07.712268 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:07.704642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42231 (local address 127.0.98.2:44155)
0926 09:35:07.704817 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:07.704823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:07.704838 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:35:07.704910 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:07.704913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:07.704936 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:07.705009 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:07.705513 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:07.706214 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:07.706957 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:07.707250 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:07.707306 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:07.709450 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:07.709458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:07.709458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:07.709464 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:07.711127 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:07.711577 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:07.711580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:07.711580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:07.711637 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:07.711808 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:07.711808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:07.711809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:07.711946 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:07.712006 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:07.712132 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 09:35:07.712200 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260926 09:35:07.712286 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:07.704545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44155 (local address 127.0.98.3:42231)
0926 09:35:07.704751 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:07.704771 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:07.704844 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:07.705037 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:07.705045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:07.705373 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:07.705438 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:07.705440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:07.706284 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:07.706285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:07.706884 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:07.706885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:07.707371 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:07.707372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:07.707458 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:07.707922 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:07.707927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:07.709361 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:07.711227 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:07.711233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:07.711235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:07.711505 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:07.711705 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:07.711706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:07.711706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:07.711761 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:07.711998 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:07.712004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:07.712048 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 09:35:07.712112 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:35:07.712206 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":119,"threads_started":1}
I20260926 09:35:07.712695 15482 tablet_copy_service.cc:140] P 994dd2b9231e4ce3b432c340226989a3: Received BeginTabletCopySession request for tablet 31cc30c179b545b8870a8ba623793cdd from peer a68137e848ee4643b02ee70de19c9dcc ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42231)
I20260926 09:35:07.712757 15482 tablet_copy_service.cc:161] P 994dd2b9231e4ce3b432c340226989a3: Beginning new tablet copy session on tablet 31cc30c179b545b8870a8ba623793cdd from peer a68137e848ee4643b02ee70de19c9dcc at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42231: session id = a68137e848ee4643b02ee70de19c9dcc-31cc30c179b545b8870a8ba623793cdd
I20260926 09:35:07.713181 15482 tablet_copy_source_session.cc:205] T 31cc30c179b545b8870a8ba623793cdd P 994dd2b9231e4ce3b432c340226989a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 09:35:07.714100 15662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc30c179b545b8870a8ba623793cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:07.716766 15662 tablet_copy_client.cc:791] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: tablet copy: Starting download of 0 data blocks...
I20260926 09:35:07.716935 15662 tablet_copy_client.cc:655] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: tablet copy: Starting download of 1 WAL segments...
I20260926 09:35:07.717761 15662 tablet_copy_client.cc:531] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 09:35:07.718447 15662 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Bootstrap starting.
I20260926 09:35:07.719789 15662 log.cc:824] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:07.719835 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:07.719711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:37981)
0926 09:35:07.719763 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:07.719809 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:35:07.722430 15662 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:07.722662 15662 tablet_bootstrap.cc:492] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Bootstrap complete.
I20260926 09:35:07.722800 15662 ts_tablet_manager.cc:1397] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 09:35:07.723840 15662 raft_consensus.cc:348] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:07.724005 15662 raft_consensus.cc:729] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68137e848ee4643b02ee70de19c9dcc, State: Initialized, Role: FOLLOWER
I20260926 09:35:07.724140 15662 consensus_queue.cc:260] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68137e848ee4643b02ee70de19c9dcc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } } peers { permanent_uuid: "994dd2b9231e4ce3b432c340226989a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44155 } } peers { permanent_uuid: "c273af5c39254444ae0f2080092c18ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } }
I20260926 09:35:07.724401 15662 ts_tablet_manager.cc:1428] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260926 09:35:07.724664 15482 tablet_copy_service.cc:342] P 994dd2b9231e4ce3b432c340226989a3: Request end of tablet copy session a68137e848ee4643b02ee70de19c9dcc-31cc30c179b545b8870a8ba623793cdd received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42231
I20260926 09:35:07.724694 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:07.724590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:50741)
0926 09:35:07.724637 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:07.724672 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:35:07.724730 15482 tablet_copy_service.cc:433] P 994dd2b9231e4ce3b432c340226989a3: ending tablet copy session a68137e848ee4643b02ee70de19c9dcc-31cc30c179b545b8870a8ba623793cdd on tablet 31cc30c179b545b8870a8ba623793cdd with peer a68137e848ee4643b02ee70de19c9dcc
I20260926 09:35:07.725016 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:07.724928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:34065)
0926 09:35:07.724968 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:07.724992 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 09:35:08.318310 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:08.318152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:58145)
0926 09:35:08.318236 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:08.318263 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":43,"threads_started":1}
I20260926 09:35:08.318307 15597 raft_consensus.cc:1203] T 31cc30c179b545b8870a8ba623793cdd P a68137e848ee4643b02ee70de19c9dcc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 09:35:08.319001 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:08.318809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:33807)
0926 09:35:08.318921 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:08.318956 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260926 09:35:08.324625 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:08.324453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:58421)
0926 09:35:08.324549 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:08.324588 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260926 09:35:09.319115 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:09.318915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:39505)
0926 09:35:09.319036 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:09.319067 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260926 09:35:09.319782 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:09.319528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:38559)
0926 09:35:09.319702 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:09.319740 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260926 09:35:09.325196 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:09.325028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:36323)
0926 09:35:09.325129 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:09.325155 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
I20260926 09:35:10.319949 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:10.319726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:45651)
0926 09:35:10.319862 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:10.319902 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260926 09:35:10.320485 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:10.320261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:33071)
0926 09:35:10.320383 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:10.320416 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260926 09:35:10.325875 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:10.325691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:53071)
0926 09:35:10.325801 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:10.325835 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260926 09:35:11.320726 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:11.320481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:58657)
0926 09:35:11.320600 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:11.320641 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260926 09:35:11.321135 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:11.321048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:48165)
0926 09:35:11.321093 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:11.321117 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:35:11.321133 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:11.320944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:56157)
0926 09:35:11.321052 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:11.321085 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":47,"threads_started":1}
I20260926 09:35:11.326618 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:11.326434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:44187)
0926 09:35:11.326550 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:11.326583 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260926 09:35:12.321792 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:12.321582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:44713)
0926 09:35:12.321710 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:12.321744 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260926 09:35:12.321821 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:12.321584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:41549)
0926 09:35:12.321738 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:12.321767 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
I20260926 09:35:12.327283 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:12.327042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:45417)
0926 09:35:12.327194 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:12.327229 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260926 09:35:13.322587 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:13.322344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:46425)
0926 09:35:13.322495 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:13.322541 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260926 09:35:13.322606 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:13.322344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:33075)
0926 09:35:13.322526 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:13.322563 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260926 09:35:13.328182 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:13.327983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:51819)
0926 09:35:13.328099 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:13.328138 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260926 09:35:14.323292 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:14.323056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:52241)
0926 09:35:14.323197 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:14.323233 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260926 09:35:14.323398 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:14.323148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:41827)
0926 09:35:14.323274 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:14.323304 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260926 09:35:14.328876 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:14.328698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:40607)
0926 09:35:14.328800 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:14.328839 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
I20260926 09:35:15.324055 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:15.323819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:40823)
0926 09:35:15.323962 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:15.324004 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260926 09:35:15.324088 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:15.323872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:34291)
0926 09:35:15.324014 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:15.324048 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260926 09:35:15.329450 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:15.329287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:56707)
0926 09:35:15.329386 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:15.329411 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
I20260926 09:35:16.324805 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:16.324554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:46459)
0926 09:35:16.324691 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:16.324739 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
I20260926 09:35:16.324869 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:16.324656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:48157)
0926 09:35:16.324793 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:16.324828 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260926 09:35:16.330080 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:16.329877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:36919)
0926 09:35:16.330003 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:16.330039 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":53,"threads_started":1}
I20260926 09:35:17.325518 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:17.325304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:46295)
0926 09:35:17.325434 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:17.325471 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260926 09:35:17.325534 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:17.325304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:44615)
0926 09:35:17.325458 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:17.325493 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260926 09:35:17.330670 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:17.330493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:52561)
0926 09:35:17.330596 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:17.330631 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260926 09:35:18.326229 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:18.325980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:55985)
0926 09:35:18.326118 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:18.326160 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260926 09:35:18.326237 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:18.325979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:46561)
0926 09:35:18.326139 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:18.326178 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260926 09:35:18.331236 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:18.331044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:48347)
0926 09:35:18.331164 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:18.331196 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260926 09:35:19.326994 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:19.326768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:41069)
0926 09:35:19.326901 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:19.326946 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260926 09:35:19.327014 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:19.326790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:39297)
0926 09:35:19.326940 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:19.326974 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260926 09:35:19.331962 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:19.331728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:58111)
0926 09:35:19.331876 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:19.331917 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260926 09:35:20.327672 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:20.327470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.3:49715)
0926 09:35:20.327596 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:20.327625 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260926 09:35:20.327687 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:20.327482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.2:43231)
0926 09:35:20.327613 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:20.327646 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260926 09:35:20.332569 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:20.332381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38091 (local address 127.0.98.1:40747)
0926 09:35:20.332494 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:20.332530 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
I20260926 09:35:20.726125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 15214
I20260926 09:35:20.730813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 15378
I20260926 09:35:20.735517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 15514
2026-09-26T09:35:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18310 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:35:20 dist-test-slave-21h8 krb5kdc[15713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:35:20 dist-test-slave-21h8 krb5kdc[15713](info): set up 2 sockets
Sep 26 09:35:20 dist-test-slave-21h8 krb5kdc[15713](info): commencing operation
krb5kdc: starting...
W20260926 09:35:21.787529 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:35:21 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:35:21.903298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37461
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:21.984258 15726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:21.984470 15726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:21.984491 15726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:21.986083 15726 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:35:21.986124 15726 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:35:21.986137 15726 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:35:21.986150 15726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:35:21.986162 15726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:21.986176 15726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:21.986188 15726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:35:21.986208 15726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:35:21.987717 15726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:21.987902 15726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:21.988062 15726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:21.992753 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:21.992763 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:21.992792 15726 server_base.cc:1034] running on GCE node
W20260926 09:35:21.992761 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:21.993191 15726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:21.993490 15726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:21.994658 15726 hybrid_clock.cc:648] HybridClock initialized: now 1790415321994636 us; error 35 us; skew 500 ppm
Sep 26 09:35:21 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:21.997102 15726 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:35:21.997561 15726 webserver.cc:460] Webserver started at http://127.0.98.62:44655/ using document root <none> and password file <none>
I20260926 09:35:21.997786 15726 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:21.997861 15726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:21.998003 15726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:21.998993 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "382bd0779d1942b799bf8921c3db8275"
format_stamp: "Formatted at 2026-09-26 09:35:21 on dist-test-slave-21h8"
server_key: "870bd2580e0c42bf7e98e1cae75534c2"
server_key_iv: "3c6163a8a6dcdc454432096c15855e00"
server_key_version: "encryptionkey@0"
I20260926 09:35:21.999434 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "382bd0779d1942b799bf8921c3db8275"
format_stamp: "Formatted at 2026-09-26 09:35:21 on dist-test-slave-21h8"
server_key: "870bd2580e0c42bf7e98e1cae75534c2"
server_key_iv: "3c6163a8a6dcdc454432096c15855e00"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.000715 15726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:22.001376 15741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.001523 15726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:35:22.001621 15726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "382bd0779d1942b799bf8921c3db8275"
format_stamp: "Formatted at 2026-09-26 09:35:21 on dist-test-slave-21h8"
server_key: "870bd2580e0c42bf7e98e1cae75534c2"
server_key_iv: "3c6163a8a6dcdc454432096c15855e00"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.001701 15726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:22.013535 15726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:22.015196 15726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:22.015452 15726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:22.019446 15726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37461
I20260926 09:35:22.019495 15793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37461 every 8 connection(s)
I20260926 09:35:22.019873 15726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:35:22.020685 15794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:22.022872 15794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Bootstrap starting.
I20260926 09:35:22.023487 15794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:22.023792 15794 log.cc:824] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:22.024467 15794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: No bootstrap required, opened a new log
I20260926 09:35:22.026014 15794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:35:22.026149 15794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:22.026170 15794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382bd0779d1942b799bf8921c3db8275, State: Initialized, Role: FOLLOWER
I20260926 09:35:22.026337 15794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:35:22.026407 15794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:35:22.026432 15794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:35:22.026473 15794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:22.027002 15794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:35:22.027117 15794 leader_election.cc:302] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382bd0779d1942b799bf8921c3db8275; no voters:
I20260926 09:35:22.027304 15794 leader_election.cc:288] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:35:22.027417 15797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:35:22.027556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 15726
I20260926 09:35:22.027655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:35:22.027624 15794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:35:22.027623 15797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 1 LEADER]: Becoming Leader. State: Replica: 382bd0779d1942b799bf8921c3db8275, State: Running, Role: LEADER
I20260926 09:35:22.027850 392 external_mini_cluster.cc:1428] Setting key ad21f8722426689554b2cbe0cd7f1ee8
I20260926 09:35:22.027745 15797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:35:22.028517 15799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "382bd0779d1942b799bf8921c3db8275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } }
I20260926 09:35:22.028617 15799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:22.028544 15798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382bd0779d1942b799bf8921c3db8275. Latest consensus state: current_term: 1 leader_uuid: "382bd0779d1942b799bf8921c3db8275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } }
I20260926 09:35:22.028730 15798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:22.028926 15809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:35:22.029420 15809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:35:22.031872 15809 catalog_manager.cc:1347] Generated new cluster ID: e266de89e3db48be94e94234821d0321
I20260926 09:35:22.031934 15809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:35:22.031963 15816 mvcc.cc:205] Tried to move back new op lower bound from 7333541159035645952 to 7333541159026864128. Current Snapshot: MvccSnapshot[applied={T|T < 7333541159035645952 or (T in {7333541159035645952})}]
Sep 26 09:35:22 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:22.038820 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.028339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.0.98.62:37461)
0926 09:35:22.028833 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.028855 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.028890 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:35:22.029006 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.029012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.029066 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.029882 (+ 816us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:22.029954 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.030850 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.031915 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.032348 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.032430 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.035171 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:22.035175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:22.035175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:22.035191 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:22.037101 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.037516 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.037519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.037519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.037637 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.037862 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.037863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.037863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.038151 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:22.038276 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.038463 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.038595 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":91,"threads_started":1}
I20260926 09:35:22.043596 15809 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:35:22.044090 15809 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:35:22.050519 15809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Generated new TSK 0
I20260926 09:35:22.050693 15809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:35:22.081821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:22.162132 15820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:22.162334 15820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:22.162356 15820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:22.163548 15820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:22.163592 15820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:22.163618 15820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:35:22.165019 15820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:22.165201 15820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:22.165385 15820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:22.170272 15828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:22.170420 15825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:22.170392 15826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:22.170394 15820 server_base.cc:1034] running on GCE node
I20260926 09:35:22.170693 15820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:22.170895 15820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:22.172039 15820 hybrid_clock.cc:648] HybridClock initialized: now 1790415322172031 us; error 29 us; skew 500 ppm
Sep 26 09:35:22 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:22.174911 15820 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:35:22.175449 15820 webserver.cc:460] Webserver started at http://127.0.98.1:39801/ using document root <none> and password file <none>
I20260926 09:35:22.175745 15820 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:22.175812 15820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:22.175951 15820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:22.176986 15820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "df3b9993cde4465a9419e4d5c05e9178"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "fa867719a395a752d37d43f29c544490"
server_key_iv: "7fc9b2c1f3999282c7a5a90fea768758"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.177362 15820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "df3b9993cde4465a9419e4d5c05e9178"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "fa867719a395a752d37d43f29c544490"
server_key_iv: "7fc9b2c1f3999282c7a5a90fea768758"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.178670 15820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:35:22.179450 15835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.179622 15820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:22.179719 15820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "df3b9993cde4465a9419e4d5c05e9178"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "fa867719a395a752d37d43f29c544490"
server_key_iv: "7fc9b2c1f3999282c7a5a90fea768758"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.179806 15820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:22.199600 15820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:22.201483 15820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:22.201653 15820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:22.201905 15820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:22.202220 15820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:22.202294 15820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.202346 15820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:22.202380 15820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.208772 15820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43797
I20260926 09:35:22.208841 15948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43797 every 8 connection(s)
I20260926 09:35:22.209183 15820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:35:22 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:22.216344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 15820
I20260926 09:35:22.216454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:35:22.216650 392 external_mini_cluster.cc:1428] Setting key d0ac5d3389bf8d78f95769d8b67e6eba
I20260926 09:35:22.218286 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.209950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47565 (local address 127.0.98.62:37461)
0926 09:35:22.210022 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.210024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.210244 (+ 220us) server_negotiation.cc:407] Connection header received
0926 09:35:22.210711 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.210712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.210726 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.210777 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:22.211799 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.212293 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.213214 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.213408 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.213458 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.215451 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:22.215453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:22.215453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:22.215459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:22.216860 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.217311 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.217312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.217312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.217387 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.217674 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.217675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.217675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.217775 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:22.217827 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.218156 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.218223 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:35:22.218588 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.209938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:47565)
0926 09:35:22.210117 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.210155 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:22.210628 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:22.210835 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:22.210838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:22.211511 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:22.211719 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.211724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.212370 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.212370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.213137 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.213139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.213479 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.213479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.213577 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.214020 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:22.214027 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:22.215372 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:22.216956 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:22.216963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:22.216966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:22.217234 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:22.217460 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:22.217461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:22.217461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:22.217602 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:22.217840 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:22.217846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:22.218019 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 09:35:22.218135 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 09:35:22.218318 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260926 09:35:22.219102 15949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260926 09:35:22.219216 15949 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:22.219437 15949 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260926 09:35:22.220014 15758 ts_manager.cc:194] Registered new tserver with Master: df3b9993cde4465a9419e4d5c05e9178 (127.0.98.1:43797)
I20260926 09:35:22.220616 15758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47565
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:35:22.241937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:22.323105 15954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:22.323305 15954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:22.323352 15954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:22.324434 15954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:22.324469 15954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:22.324496 15954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:35:22.325861 15954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:22.326048 15954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:22.326215 15954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:22.331261 15959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:22.331287 15962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:22.331370 15960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:22.331391 15954 server_base.cc:1034] running on GCE node
I20260926 09:35:22.331722 15954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:22.331988 15954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:22.333123 15954 hybrid_clock.cc:648] HybridClock initialized: now 1790415322333112 us; error 28 us; skew 500 ppm
Sep 26 09:35:22 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:22.335595 15954 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:35:22.336023 15954 webserver.cc:460] Webserver started at http://127.0.98.2:39975/ using document root <none> and password file <none>
I20260926 09:35:22.336213 15954 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:22.336254 15954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:22.336347 15954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:22.337312 15954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "0177fc1894daf74f744beccd4909b86c"
server_key_iv: "d280a76f7ab9a8157c5265eeaa064e59"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.337685 15954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "0177fc1894daf74f744beccd4909b86c"
server_key_iv: "d280a76f7ab9a8157c5265eeaa064e59"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.338937 15954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 09:35:22.339661 15969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.339821 15954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.339900 15954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "0177fc1894daf74f744beccd4909b86c"
server_key_iv: "d280a76f7ab9a8157c5265eeaa064e59"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.339952 15954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:22.349920 15954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:22.351701 15954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:22.351814 15954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:22.351976 15954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:22.352211 15954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:22.352247 15954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.352269 15954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:22.352284 15954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.357892 15954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35309
I20260926 09:35:22.357967 16082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35309 every 8 connection(s)
I20260926 09:35:22.358259 15954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:35:22 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:22.366463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 15954
I20260926 09:35:22.366565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:35:22.366793 392 external_mini_cluster.cc:1428] Setting key 2b5dd632bef0dd655e61c6e763239246
I20260926 09:35:22.366791 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.358998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55005 (local address 127.0.98.62:37461)
0926 09:35:22.359072 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.359074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.359276 (+ 202us) server_negotiation.cc:407] Connection header received
0926 09:35:22.359792 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.359793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.359803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.359859 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:22.360740 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.361225 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.362144 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.362319 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.362374 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.364403 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:22.364405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:22.364405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:22.364411 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:22.365571 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.365924 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.365925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.365925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.365971 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.366171 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.366171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.366171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.366286 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:22.366371 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.366606 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.366697 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:35:22.366943 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.358977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:55005)
0926 09:35:22.359160 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.359194 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:22.359705 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:22.359921 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:22.359924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:22.360484 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:22.360665 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.360669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.361304 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.361305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.362063 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.362064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.362396 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.362397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.362489 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.362901 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:22.362909 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:22.364322 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:22.365640 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:22.365645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:22.365647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:22.365859 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:22.366029 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:22.366029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:22.366030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:22.366118 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:22.366358 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:22.366362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:22.366463 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 09:35:22.366568 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 09:35:22.366715 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260926 09:35:22.367475 16083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260926 09:35:22.367576 16083 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:22.367743 16083 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260926 09:35:22.368145 15758 ts_manager.cc:194] Registered new tserver with Master: c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
I20260926 09:35:22.368664 15758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55005
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:35:22.390254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:22.472067 16088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:22.472311 16088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:22.472352 16088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:22.473489 16088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:22.473557 16088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:22.473601 16088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:35:22.475039 16088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:22.475338 16088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:22.475582 16088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:22.480882 16096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:22.480943 16093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:22.480890 16094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:22.481025 16088 server_base.cc:1034] running on GCE node
I20260926 09:35:22.481317 16088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:22.481616 16088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:22.482787 16088 hybrid_clock.cc:648] HybridClock initialized: now 1790415322482755 us; error 32 us; skew 500 ppm
Sep 26 09:35:22 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:22.485250 16088 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:35:22.485668 16088 webserver.cc:460] Webserver started at http://127.0.98.3:37039/ using document root <none> and password file <none>
I20260926 09:35:22.485904 16088 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:22.485971 16088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:22.486102 16088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:35:22.487116 16088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "b969be9f7436655b51d0d0cf8e423eba"
server_key_iv: "479578228457755c33f9579175c1bd5b"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.487511 16088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "b969be9f7436655b51d0d0cf8e423eba"
server_key_iv: "479578228457755c33f9579175c1bd5b"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.488814 16088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:22.489547 16103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.489737 16088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:35:22.489841 16088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "b969be9f7436655b51d0d0cf8e423eba"
server_key_iv: "479578228457755c33f9579175c1bd5b"
server_key_version: "encryptionkey@0"
I20260926 09:35:22.489928 16088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:22.505785 16088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:22.508096 16088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:22.508268 16088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:22.508489 16088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:22.508796 16088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:22.508860 16088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.508901 16088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:22.508929 16088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:22.514650 16088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43329
I20260926 09:35:22.514683 16216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43329 every 8 connection(s)
I20260926 09:35:22.515036 16088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:35:22 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:22.523747 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.515713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43941 (local address 127.0.98.62:37461)
0926 09:35:22.515774 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.515776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.516038 (+ 262us) server_negotiation.cc:407] Connection header received
0926 09:35:22.516557 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.516558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.516568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.516629 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:22.517503 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.517995 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.518939 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.519111 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.519160 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.521145 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:22.521146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:22.521146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:22.521153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:22.522531 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.522878 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.522879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.522879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.522928 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.523130 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.523130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.523131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.523228 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:22.523278 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.523620 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.523683 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:35:22.523957 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.515700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:43941)
0926 09:35:22.515880 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.515921 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:22.516452 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:22.516685 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:22.516688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:22.517240 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:22.517426 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.517430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.518071 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.518071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.518860 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.518862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.519182 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.519182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.519278 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.519739 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:22.519747 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:22.521059 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:22.522598 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:22.522602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:22.522604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:22.522806 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:22.522985 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:22.522986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:22.522986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:22.523077 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:22.523282 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:22.523285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:22.523469 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 09:35:22.523583 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:35:22.523705 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260926 09:35:22.524544 16217 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260926 09:35:22.524645 16217 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:22.524837 16217 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260926 09:35:22.524973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 16088
I20260926 09:35:22.525089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:35:22.525254 15758 ts_manager.cc:194] Registered new tserver with Master: 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329)
I20260926 09:35:22.525314 392 external_mini_cluster.cc:1428] Setting key 934394b55e1c4f717bfafae5a4681490
I20260926 09:35:22.525724 15758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43941
I20260926 09:35:22.526247 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:35:22.533455 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.527649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.0.98.62:37461)
0926 09:35:22.527690 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.527691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.527828 (+ 137us) server_negotiation.cc:407] Connection header received
0926 09:35:22.527893 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.527894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.527902 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.527963 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:22.528623 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.529175 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.529933 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.530104 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.530152 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.531200 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:22.531201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:22.531201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:22.531206 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:22.532574 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.532866 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.532867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.532867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.532914 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:22.533079 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:22.533079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:22.533079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:22.533191 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:22.533258 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.533345 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.533399 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:35:22.535451 15758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:35:22.537717 15758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:35:22.543709 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.539804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35309 (local address 127.0.0.1:46864)
0926 09:35:22.540090 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.540130 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:22.540199 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:22.540492 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:22.540495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:22.540497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:22.540581 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.540583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.541556 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.541557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.542619 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.542621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.543373 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.543374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.543469 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.543484 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:35:22.543584 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 09:35:22.543637 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":54,"threads_started":1}
I20260926 09:35:22.543721 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.539915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.0.98.2:35309)
0926 09:35:22.540324 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.540344 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.540353 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:35:22.540383 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.540385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.540397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.540487 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:22.540661 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.541458 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.542700 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.543176 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.543351 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.543418 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.543601 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.543642 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":94,"threads_started":1}
I20260926 09:35:22.544507 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.539993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43329 (local address 127.0.0.1:58512)
0926 09:35:22.540137 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.540147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:22.540199 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:22.541036 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:22.541037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:22.541039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:22.541078 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.541080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.542252 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.542253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.543300 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.543302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.544268 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.544269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.544344 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.544347 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:35:22.544401 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:35:22.544443 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":37,"threads_started":1}
I20260926 09:35:22.544543 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.540141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.0.98.3:43329)
0926 09:35:22.540866 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.540871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.540880 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:35:22.540916 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.540918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.540931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.541013 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:22.541142 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.542145 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.543410 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.544112 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.544187 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.544232 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.544425 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.544465 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":74,"threads_started":1}
I20260926 09:35:22.544553 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.539906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.98.1:43797)
0926 09:35:22.540091 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.540558 (+ 467us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.540587 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:35:22.540632 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.540635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.540652 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.540722 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:22.540821 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.541933 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.543461 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.544139 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.544181 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.544231 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.544435 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.544471 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260926 09:35:22.544742 16017 tablet_service.cc:1467] Processing CreateTablet for tablet 02333839248245369ae36bfec7b5807f (DEFAULT_TABLE table=test-table [id=dfed842a4dcf40eaaadc9ee6e8951859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:22.544754 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.539806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43797 (local address 127.0.0.1:43664)
0926 09:35:22.540100 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.540132 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:22.540201 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:22.540716 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:22.540717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:22.540718 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:22.540758 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.540759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.542057 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.542057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.543288 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.543290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.544287 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.544288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.544340 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.544343 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:35:22.544625 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 09:35:22.544673 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"spinlock_wait_cycles":896,"thread_start_us":78,"threads_started":1}
I20260926 09:35:22.545073 16017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02333839248245369ae36bfec7b5807f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:22.547629 16231 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Bootstrap starting.
I20260926 09:35:22.548115 16151 tablet_service.cc:1467] Processing CreateTablet for tablet 02333839248245369ae36bfec7b5807f (DEFAULT_TABLE table=test-table [id=dfed842a4dcf40eaaadc9ee6e8951859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:22.548188 15883 tablet_service.cc:1467] Processing CreateTablet for tablet 02333839248245369ae36bfec7b5807f (DEFAULT_TABLE table=test-table [id=dfed842a4dcf40eaaadc9ee6e8951859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:22.548403 16231 tablet_bootstrap.cc:654] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:22.548475 16151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02333839248245369ae36bfec7b5807f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:22.548527 15883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02333839248245369ae36bfec7b5807f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:22.548990 16231 log.cc:824] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:22.549790 16231 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: No bootstrap required, opened a new log
I20260926 09:35:22.549853 16231 ts_tablet_manager.cc:1397] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:22.551860 16234 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Bootstrap starting.
I20260926 09:35:22.552035 16233 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Bootstrap starting.
I20260926 09:35:22.551944 16231 raft_consensus.cc:348] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.552120 16231 raft_consensus.cc:374] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:22.552189 16231 raft_consensus.cc:729] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93a8c5c4b8e46fb9d5ee08ebb623487, State: Initialized, Role: FOLLOWER
I20260926 09:35:22.552358 16231 consensus_queue.cc:260] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.552599 16234 tablet_bootstrap.cc:654] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:22.552655 16231 ts_tablet_manager.cc:1428] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:35:22.552709 16233 tablet_bootstrap.cc:654] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:22.552793 16083 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260926 09:35:22.553227 16234 log.cc:824] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:22.553295 16233 log.cc:824] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:22.553987 16234 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: No bootstrap required, opened a new log
I20260926 09:35:22.554056 16234 ts_tablet_manager.cc:1397] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:22.554157 16233 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: No bootstrap required, opened a new log
I20260926 09:35:22.554291 16233 ts_tablet_manager.cc:1397] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:22.555557 16234 raft_consensus.cc:348] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.555716 16234 raft_consensus.cc:374] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:22.555761 16234 raft_consensus.cc:729] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b9993cde4465a9419e4d5c05e9178, State: Initialized, Role: FOLLOWER
I20260926 09:35:22.555677 16233 raft_consensus.cc:348] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.555846 16233 raft_consensus.cc:374] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:22.555897 16233 raft_consensus.cc:729] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ccf2da49cd941adb8feb6d1e6d3ec6a, State: Initialized, Role: FOLLOWER
I20260926 09:35:22.555886 16234 consensus_queue.cc:260] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.556017 16233 consensus_queue.cc:260] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.556174 16234 ts_tablet_manager.cc:1428] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:22.556195 15949 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260926 09:35:22.556327 16233 ts_tablet_manager.cc:1428] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:22.556373 16217 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
W20260926 09:35:22.608814 16084 tablet.cc:2369] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:35:22.709990 15950 tablet.cc:2369] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:35:22.748966 16239 raft_consensus.cc:482] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:35:22.749133 16239 raft_consensus.cc:504] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.750257 16239 leader_election.cc:288] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309), df3b9993cde4465a9419e4d5c05e9178 (127.0.98.1:43797)
I20260926 09:35:22.753053 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.749806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35309 (local address 127.0.98.3:47539)
0926 09:35:22.749921 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.749934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:22.750007 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:22.750193 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:22.750195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:22.750196 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:22.750271 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.750273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.750934 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.750936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.751931 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.751933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.752785 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.752785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.752877 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.752880 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:35:22.752936 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:35:22.752980 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 09:35:22.753053 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.749856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47539 (local address 127.0.98.2:35309)
0926 09:35:22.749917 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.749919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.750035 (+ 116us) server_negotiation.cc:407] Connection header received
0926 09:35:22.750068 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.750069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.750083 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.750137 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:22.750330 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.750849 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.752027 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.752705 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.752797 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.752810 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.752946 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.752991 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:35:22.753665 16037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02333839248245369ae36bfec7b5807f" candidate_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" is_pre_election: true
I20260926 09:35:22.753826 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.749993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43797 (local address 127.0.98.3:36825)
0926 09:35:22.750169 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.750179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:22.750218 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:22.750624 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:22.750625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:22.750627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:22.750717 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.750719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.751725 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.751726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.752598 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:22.752600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.753580 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.753581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:22.753647 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.753653 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:35:22.753715 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 09:35:22.753751 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":114,"threads_started":1}
I20260926 09:35:22.753847 16037 raft_consensus.cc:2454] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ccf2da49cd941adb8feb6d1e6d3ec6a in term 0.
I20260926 09:35:22.753857 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.750381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36825 (local address 127.0.98.1:43797)
0926 09:35:22.750438 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.750441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.750453 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:35:22.750487 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.750488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.750503 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.750571 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:22.750822 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.751625 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.752849 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.753485 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.753560 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.753571 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.753731 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.753762 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:35:22.754082 16104 leader_election.cc:302] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ccf2da49cd941adb8feb6d1e6d3ec6a, c93a8c5c4b8e46fb9d5ee08ebb623487; no voters:
I20260926 09:35:22.754220 16239 raft_consensus.cc:2781] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:35:22.754274 16239 raft_consensus.cc:482] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:35:22.754321 16239 raft_consensus.cc:3037] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:22.754482 15903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02333839248245369ae36bfec7b5807f" candidate_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3b9993cde4465a9419e4d5c05e9178" is_pre_election: true
I20260926 09:35:22.754632 15903 raft_consensus.cc:2454] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ccf2da49cd941adb8feb6d1e6d3ec6a in term 0.
I20260926 09:35:22.755123 16239 raft_consensus.cc:504] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.755252 16239 leader_election.cc:288] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [CANDIDATE]: Term 1 election: Requested vote from peers c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309), df3b9993cde4465a9419e4d5c05e9178 (127.0.98.1:43797)
I20260926 09:35:22.755430 15903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02333839248245369ae36bfec7b5807f" candidate_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3b9993cde4465a9419e4d5c05e9178"
I20260926 09:35:22.755506 15903 raft_consensus.cc:3037] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:22.756196 15903 raft_consensus.cc:2454] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ccf2da49cd941adb8feb6d1e6d3ec6a in term 1.
I20260926 09:35:22.756315 16037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02333839248245369ae36bfec7b5807f" candidate_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487"
I20260926 09:35:22.756358 16106 leader_election.cc:302] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ccf2da49cd941adb8feb6d1e6d3ec6a, df3b9993cde4465a9419e4d5c05e9178; no voters:
I20260926 09:35:22.756408 16037 raft_consensus.cc:3037] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:22.756460 16239 raft_consensus.cc:2781] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:35:22.756603 16239 raft_consensus.cc:686] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 LEADER]: Becoming Leader. State: Replica: 9ccf2da49cd941adb8feb6d1e6d3ec6a, State: Running, Role: LEADER
I20260926 09:35:22.756697 16239 consensus_queue.cc:237] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:22.757285 16037 raft_consensus.cc:2454] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ccf2da49cd941adb8feb6d1e6d3ec6a in term 1.
I20260926 09:35:22.757872 15758 catalog_manager.cc:5495] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } health_report { overall_health: HEALTHY } } }
W20260926 09:35:22.765533 16218 tablet.cc:2369] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:35:22.775390 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.773259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.0.98.3:43329)
0926 09:35:22.773300 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:22.773301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:22.773309 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:35:22.773379 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:22.773380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:22.773397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:22.773456 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:35:22.773561 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.774009 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.774710 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:22.774890 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:22.774939 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:22.775062 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:35:22.775156 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:35:22.775209 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:22.775243 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 09:35:22.775262 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:35:22.777132 15903 raft_consensus.cc:1261] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Refusing update from remote peer 9ccf2da49cd941adb8feb6d1e6d3ec6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:35:22.777184 16037 raft_consensus.cc:1261] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Refusing update from remote peer 9ccf2da49cd941adb8feb6d1e6d3ec6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:35:22.777529 16241 consensus_queue.cc:1035] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:35:22.777657 16239 consensus_queue.cc:1035] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:35:22.779922 16244 mvcc.cc:205] Tried to move back new op lower bound from 7333541162092511232 to 7333541162012078080. Current Snapshot: MvccSnapshot[applied={T|T < 7333541162092511232}]
I20260926 09:35:22.782725 15758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:35:22.782827 15758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:35:22.783613 15758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Sending DeleteTablet for 3 replicas of tablet 02333839248245369ae36bfec7b5807f
I20260926 09:35:22.784018 16017 tablet_service.cc:1514] Processing DeleteTablet for tablet 02333839248245369ae36bfec7b5807f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46864
I20260926 09:35:22.784049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 15820
I20260926 09:35:22.784216 16151 tablet_service.cc:1514] Processing DeleteTablet for tablet 02333839248245369ae36bfec7b5807f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58512
I20260926 09:35:22.784698 16253 tablet_replica.cc:330] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: stopping tablet replica
I20260926 09:35:22.784787 16253 raft_consensus.cc:2229] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:22.784876 16253 raft_consensus.cc:2258] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 09:35:22.788874 15744 connection.cc:535] client connection to 127.0.98.1:43797 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 09:35:22.788959 15744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:35:22.789069 16254 tablet_replica.cc:330] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: stopping tablet replica
I20260926 09:35:22.789163 16254 raft_consensus.cc:2229] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:35:22.789294 16254 raft_consensus.cc:2258] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 09:35:22.789557 16225 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:35:22.789431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43797 (local address 127.0.0.1:43670)
0926 09:35:22.789480 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.789518 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43797: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:35:22.789685 15744 catalog_manager.cc:4584] TS df3b9993cde4465a9419e4d5c05e9178 (127.0.98.1:43797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02333839248245369ae36bfec7b5807f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43797: connect: Connection reset by peer (error 104)
I20260926 09:35:22.789893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 15954
I20260926 09:35:22.790896 16254 ts_tablet_manager.cc:1905] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:22.792788 16254 ts_tablet_manager.cc:1918] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:22.792892 16254 log.cc:1196] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/02333839248245369ae36bfec7b5807f
I20260926 09:35:22.793186 16254 ts_tablet_manager.cc:1939] T 02333839248245369ae36bfec7b5807f P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Deleting consensus metadata
I20260926 09:35:22.793649 15744 catalog_manager.cc:4841] TS 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329): tablet 02333839248245369ae36bfec7b5807f (table test-table [id=dfed842a4dcf40eaaadc9ee6e8951859]) successfully deleted
I20260926 09:35:22.795255 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:22.795134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35309 (local address 127.0.0.1:46878)
0926 09:35:22.795192 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:22.795225 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:35:22.795388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 16088
I20260926 09:35:22.800316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 15726
I20260926 09:35:42.805759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37461
--webserver_interface=127.0.98.62
--webserver_port=44655
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:35:42.900661 16256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:42.900882 16256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:42.900919 16256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:42.902036 16256 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:35:42.902104 16256 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:35:42.902128 16256 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:35:42.902158 16256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:35:42.902184 16256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:42.902215 16256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:42.902240 16256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:35:42.902261 16256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:35:42.903764 16256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:42.904026 16256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:42.904258 16256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:42.912765 16265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:42.912777 16263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:42.912782 16262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:42.913023 16256 server_base.cc:1034] running on GCE node
I20260926 09:35:42.913352 16256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:42.913736 16256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:42.914916 16256 hybrid_clock.cc:648] HybridClock initialized: now 1790415342914897 us; error 43 us; skew 500 ppm
Sep 26 09:35:42 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:42.917759 16256 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:35:42.918267 16256 webserver.cc:460] Webserver started at http://127.0.98.62:44655/ using document root <none> and password file <none>
I20260926 09:35:42.918509 16256 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:42.918583 16256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:42.920120 16256 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:35:42.920835 16272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:42.921039 16256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:35:42.921172 16256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "382bd0779d1942b799bf8921c3db8275"
format_stamp: "Formatted at 2026-09-26 09:35:21 on dist-test-slave-21h8"
server_key: "870bd2580e0c42bf7e98e1cae75534c2"
server_key_iv: "3c6163a8a6dcdc454432096c15855e00"
server_key_version: "encryptionkey@0"
I20260926 09:35:42.921582 16256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:42.946664 16256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:42.948566 16256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:42.948721 16256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:42.952822 16256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37461
I20260926 09:35:42.952899 16324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37461 every 8 connection(s)
I20260926 09:35:42.953284 16256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:35:42.954766 16325 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:35:42.955454 16325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Bootstrap starting.
I20260926 09:35:42.960829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 16256
I20260926 09:35:42.961115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43797
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=39801
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:35:42.964118 16325 log.cc:824] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:42.966012 16325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:42.966297 16325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Bootstrap complete.
I20260926 09:35:42.968606 16325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:35:42.969000 16325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382bd0779d1942b799bf8921c3db8275, State: Initialized, Role: FOLLOWER
I20260926 09:35:42.969216 16325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:35:42.969305 16325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:35:42.969359 16325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:35:42.969422 16325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:35:42.970577 16325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:35:42.970695 16325 leader_election.cc:302] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382bd0779d1942b799bf8921c3db8275; no voters:
I20260926 09:35:42.970887 16325 leader_election.cc:288] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:35:42.970919 16329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:35:42.971046 16329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 2 LEADER]: Becoming Leader. State: Replica: 382bd0779d1942b799bf8921c3db8275, State: Running, Role: LEADER
I20260926 09:35:42.971190 16325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:35:42.971201 16329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:35:42.971622 16330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "382bd0779d1942b799bf8921c3db8275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } }
I20260926 09:35:42.971714 16330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:42.972043 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382bd0779d1942b799bf8921c3db8275. Latest consensus state: current_term: 2 leader_uuid: "382bd0779d1942b799bf8921c3db8275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } }
I20260926 09:35:42.972126 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: This master's current role is: LEADER
I20260926 09:35:42.972433 16337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:35:42.973191 16337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:35:42.973451 16337 catalog_manager.cc:1259] Loaded cluster ID: e266de89e3db48be94e94234821d0321
I20260926 09:35:42.973609 16337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:35:42.974790 16337 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:35:42.974968 16337 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Loaded TSK: 0
I20260926 09:35:42.986642 16337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Generated new TSK 1
I20260926 09:35:42.986733 16337 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:35:43.044430 16327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:43.044659 16327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:43.044699 16327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:43.045915 16327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:43.045977 16327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:43.046021 16327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:35:43.047535 16327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39801
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:43.047785 16327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:43.048038 16327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:43.054013 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:43.054062 16327 server_base.cc:1034] running on GCE node
W20260926 09:35:43.054028 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:43.054162 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:43.054551 16327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:43.054816 16327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:43.055974 16327 hybrid_clock.cc:648] HybridClock initialized: now 1790415343055960 us; error 32 us; skew 500 ppm
Sep 26 09:35:43 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:43.058445 16327 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:35:43.058939 16327 webserver.cc:460] Webserver started at http://127.0.98.1:39801/ using document root <none> and password file <none>
I20260926 09:35:43.059152 16327 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:43.059233 16327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:43.060590 16327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:43.061322 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:43.061501 16327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:35:43.061594 16327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "df3b9993cde4465a9419e4d5c05e9178"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "fa867719a395a752d37d43f29c544490"
server_key_iv: "7fc9b2c1f3999282c7a5a90fea768758"
server_key_version: "encryptionkey@0"
I20260926 09:35:43.061955 16327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:43.084913 16327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:43.087133 16327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:43.087296 16327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:43.087585 16327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:43.088040 16370 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:35:43.089444 16327 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:35:43.089499 16327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:35:43.089524 16327 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:35:43.090212 16327 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:35:43.090255 16327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:35:43.090299 16370 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Bootstrap starting.
I20260926 09:35:43.098079 16327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43797
I20260926 09:35:43.098146 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43797 every 8 connection(s)
I20260926 09:35:43.098798 16327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:35:43.102629 16370 log.cc:824] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:43.104904 16370 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:43.105237 16370 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Bootstrap complete.
I20260926 09:35:43.106194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 16327
I20260926 09:35:43.106315 16370 ts_tablet_manager.cc:1397] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20260926 09:35:43.106470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35309
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=39975
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:35:43 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:43.108062 16370 raft_consensus.cc:348] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:43.108431 16370 raft_consensus.cc:729] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b9993cde4465a9419e4d5c05e9178, State: Initialized, Role: FOLLOWER
I20260926 09:35:43.108605 16370 consensus_queue.cc:260] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:43.109064 16370 ts_tablet_manager.cc:1428] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260926 09:35:43.111388 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.099624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53277 (local address 127.0.98.62:37461)
0926 09:35:43.099820 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.099831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.099883 (+ 52us) server_negotiation.cc:407] Connection header received
0926 09:35:43.100563 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.100571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.100648 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.100820 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:43.101670 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.103147 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.104496 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.105105 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.105225 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.107646 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:43.107650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:43.107651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:43.107662 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:43.109644 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.110052 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.110055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.110055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.110164 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.110419 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.110420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.110420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.110647 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:43.110732 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.111030 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.111175 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260926 09:35:43.111399 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.099497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:53277)
0926 09:35:43.099700 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.099745 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:43.100334 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:43.100874 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:43.100878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:43.101269 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:43.101514 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.101520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.103287 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.103288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.104404 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.104407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.105212 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.105213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.105359 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.105917 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:43.105926 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:43.107524 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:43.109730 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.109737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.109739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.109964 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.110242 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.110243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.110243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.110351 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.110725 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:43.110732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:43.110869 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 09:35:43.110975 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:35:43.111168 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":113,"threads_started":1}
I20260926 09:35:43.112033 16478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260926 09:35:43.112124 16478 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:43.112305 16478 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260926 09:35:43.113008 16289 ts_manager.cc:194] Registered new tserver with Master: df3b9993cde4465a9419e4d5c05e9178 (127.0.98.1:43797)
I20260926 09:35:43.114009 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53277
I20260926 09:35:43.115000 16478 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
Sep 26 09:35:43 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:35:43.122510 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.113634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43797 (local address 127.0.0.1:34150)
0926 09:35:43.114321 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.114353 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:43.114411 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:43.114715 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:43.114722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:43.115100 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:43.115180 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.115183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.116177 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.116178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.117301 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.117304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.117745 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.117749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.117822 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.118248 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:43.118253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:43.119640 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:43.121375 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.121382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.121383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.121574 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.121807 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.121808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.121808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.121870 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.122147 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:43.122153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:43.122203 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 09:35:43.122296 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 09:35:43.122416 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":91,"threads_started":1}
I20260926 09:35:43.122521 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.113701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.0.98.1:43797)
0926 09:35:43.114550 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.114554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.114563 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:35:43.114602 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.114604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.114617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.114686 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:43.115268 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.116088 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.117389 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.117677 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.117726 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.119727 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:43.119734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:43.119734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:43.119741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:43.121271 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.121670 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.121673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.121674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.121732 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.121940 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.121941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.121941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.122066 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:43.122155 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.122332 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.122416 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":86,"threads_started":1}
I20260926 09:35:43.123093 16412 tablet_service.cc:1514] Processing DeleteTablet for tablet 02333839248245369ae36bfec7b5807f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34150
I20260926 09:35:43.123692 16487 tablet_replica.cc:330] stopping tablet replica
I20260926 09:35:43.123795 16487 raft_consensus.cc:2229] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:43.123863 16487 raft_consensus.cc:2258] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:43.124153 16487 ts_tablet_manager.cc:1905] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:43.126391 16487 ts_tablet_manager.cc:1918] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:43.126451 16487 log.cc:1196] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal/wals/02333839248245369ae36bfec7b5807f
I20260926 09:35:43.126696 16487 ts_tablet_manager.cc:1939] T 02333839248245369ae36bfec7b5807f P df3b9993cde4465a9419e4d5c05e9178: Deleting consensus metadata
I20260926 09:35:43.127200 16275 catalog_manager.cc:4841] TS df3b9993cde4465a9419e4d5c05e9178 (127.0.98.1:43797): tablet 02333839248245369ae36bfec7b5807f (table test-table [id=dfed842a4dcf40eaaadc9ee6e8951859]) successfully deleted
W20260926 09:35:43.189553 16483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:43.189750 16483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:43.189786 16483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:43.190872 16483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:43.190903 16483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:43.190928 16483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:35:43.192430 16483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39975
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:43.192613 16483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:43.192785 16483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:43.197613 16495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:43.197613 16493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:43.197829 16483 server_base.cc:1034] running on GCE node
W20260926 09:35:43.197881 16492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:43.198091 16483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:43.198344 16483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:43.199506 16483 hybrid_clock.cc:648] HybridClock initialized: now 1790415343199499 us; error 37 us; skew 500 ppm
Sep 26 09:35:43 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:43.201824 16483 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:35:43.202289 16483 webserver.cc:460] Webserver started at http://127.0.98.2:39975/ using document root <none> and password file <none>
I20260926 09:35:43.202522 16483 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:43.202600 16483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:43.204873 16483 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:43.205587 16502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:43.205755 16483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:35:43.205859 16483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "0177fc1894daf74f744beccd4909b86c"
server_key_iv: "d280a76f7ab9a8157c5265eeaa064e59"
server_key_version: "encryptionkey@0"
I20260926 09:35:43.206202 16483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:43.225348 16483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:43.227252 16483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:43.227468 16483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:43.227713 16483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:43.228132 16509 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:35:43.229543 16483 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:35:43.229602 16483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:35:43.229651 16483 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:35:43.230279 16483 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:35:43.230335 16483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:35:43.230389 16509 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Bootstrap starting.
I20260926 09:35:43.231959 16509 log.cc:824] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:43.234170 16509 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:35:43.234443 16509 tablet_bootstrap.cc:492] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Bootstrap complete.
I20260926 09:35:43.234622 16509 ts_tablet_manager.cc:1397] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 09:35:43.235993 16509 raft_consensus.cc:348] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:43.236249 16509 raft_consensus.cc:729] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93a8c5c4b8e46fb9d5ee08ebb623487, State: Initialized, Role: FOLLOWER
I20260926 09:35:43.236452 16509 consensus_queue.cc:260] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } } peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } }
I20260926 09:35:43.236742 16509 ts_tablet_manager.cc:1428] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260926 09:35:43.237572 16483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35309
I20260926 09:35:43.237666 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35309 every 8 connection(s)
I20260926 09:35:43.237958 16483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:35:43.241914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 16483
I20260926 09:35:43.242216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43329
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=37039
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:35:43 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:43.247391 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.238702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52779 (local address 127.0.98.62:37461)
0926 09:35:43.238784 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.238787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.238979 (+ 192us) server_negotiation.cc:407] Connection header received
0926 09:35:43.239157 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.239158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.239173 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.239224 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:43.239887 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.240399 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.241383 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.241610 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.241696 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.244559 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:43.244560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:43.244561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:43.244569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:43.245952 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.246369 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.246370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.246371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.246433 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.246694 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.246695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.246695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.246814 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:43.246884 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.247160 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.247253 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:35:43.247643 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.238622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:52779)
0926 09:35:43.238823 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.238887 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:43.239069 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:43.239289 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:43.239293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:43.239595 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:43.239776 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.239781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.240510 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.240511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.241289 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.241291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.241707 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.241708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.241845 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.242427 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:43.242438 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:43.244460 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:43.246048 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.246055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.246057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.246281 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.246497 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.246498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.246498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.246619 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.246886 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:43.246894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:43.247030 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 09:35:43.247139 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 09:35:43.247299 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260926 09:35:43.248134 16619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260926 09:35:43.248231 16619 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:43.248399 16619 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260926 09:35:43.248921 16289 ts_manager.cc:194] Registered new tserver with Master: c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
I20260926 09:35:43.249457 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52779
Sep 26 09:35:43 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 09:35:43.256996 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.249253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35309 (local address 127.0.0.1:48222)
0926 09:35:43.249325 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.249336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:43.249377 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:43.249838 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:43.249839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:43.249993 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:43.250032 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.250034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.251435 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.251435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.251909 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.251911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.252605 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.252607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.252652 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.253067 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:43.253070 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:43.254189 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:43.256063 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.256064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.256064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.256225 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.256446 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.256447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.256447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.256495 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.256779 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:43.256779 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:43.256811 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 09:35:43.256863 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:35:43.256936 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:35:43.257207 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.249299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.0.98.2:35309)
0926 09:35:43.249673 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.249680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.249691 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:35:43.249728 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.249730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.249746 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.249823 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:43.250121 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.251337 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.251989 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.252520 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.252605 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.254269 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:43.254277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:43.254277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:43.254284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:43.255989 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.256302 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.256305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.256306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.256389 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.256550 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.256551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.256552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.256716 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:43.256794 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.256902 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.256992 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":72,"threads_started":1}
I20260926 09:35:43.257508 16552 tablet_service.cc:1514] Processing DeleteTablet for tablet 02333839248245369ae36bfec7b5807f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48222
I20260926 09:35:43.258090 16624 tablet_replica.cc:330] stopping tablet replica
I20260926 09:35:43.258193 16624 raft_consensus.cc:2229] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:43.258261 16624 raft_consensus.cc:2258] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:43.258569 16624 ts_tablet_manager.cc:1905] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:43.260671 16624 ts_tablet_manager.cc:1918] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:43.260731 16624 log.cc:1196] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/02333839248245369ae36bfec7b5807f
I20260926 09:35:43.260947 16624 ts_tablet_manager.cc:1939] T 02333839248245369ae36bfec7b5807f P c93a8c5c4b8e46fb9d5ee08ebb623487: Deleting consensus metadata
I20260926 09:35:43.261431 16273 catalog_manager.cc:4841] TS c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309): tablet 02333839248245369ae36bfec7b5807f (table test-table [id=dfed842a4dcf40eaaadc9ee6e8951859]) successfully deleted
W20260926 09:35:43.326371 16622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:35:43.326647 16622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:35:43.326704 16622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:35:43.327965 16622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:35:43.328025 16622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:35:43.328071 16622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:35:43.329500 16622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37039
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:35:43.329779 16622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:35:43.330001 16622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:35:43.334839 16629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:43.334933 16622 server_base.cc:1034] running on GCE node
W20260926 09:35:43.334857 16632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:35:43.334847 16630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:35:43.335244 16622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:35:43.335544 16622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:35:43.336712 16622 hybrid_clock.cc:648] HybridClock initialized: now 1790415343336712 us; error 35 us; skew 500 ppm
Sep 26 09:35:43 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:35:43.339782 16622 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:35:43.340245 16622 webserver.cc:460] Webserver started at http://127.0.98.3:37039/ using document root <none> and password file <none>
I20260926 09:35:43.340468 16622 fs_manager.cc:359] Metadata directory not provided
I20260926 09:35:43.340524 16622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:35:43.346522 16622 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260926 09:35:43.347456 16639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:43.347738 16622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:35:43.347855 16622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "b969be9f7436655b51d0d0cf8e423eba"
server_key_iv: "479578228457755c33f9579175c1bd5b"
server_key_version: "encryptionkey@0"
I20260926 09:35:43.348330 16622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:35:43.376318 16622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:35:43.378064 16622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:35:43.378181 16622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:35:43.378355 16622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:35:43.378684 16622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:35:43.378717 16622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:43.378774 16622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:35:43.378806 16622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:35:43.384479 16622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43329
I20260926 09:35:43.384558 16752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43329 every 8 connection(s)
I20260926 09:35:43.384886 16622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:35:43.388201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 16622
Sep 26 09:35:43 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:35:43.394964 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.385602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34353 (local address 127.0.98.62:37461)
0926 09:35:43.385665 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.385667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.385864 (+ 197us) server_negotiation.cc:407] Connection header received
0926 09:35:43.386323 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.386324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.386333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.386389 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:43.387250 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.387931 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.389314 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.389521 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.389595 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.392142 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:43.392144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:43.392144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:43.392152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:43.393503 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.393898 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.393899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.393899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.393978 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.394266 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.394267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.394268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.394468 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:43.394542 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.394786 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.394861 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:35:43.395112 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.385585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:34353)
0926 09:35:43.385739 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.385780 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:43.386246 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:43.386442 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:43.386445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:43.386997 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:35:43.387180 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.387184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.388047 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.388049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.389222 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.389224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.389600 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.389601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.389748 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.390279 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:35:43.390288 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:35:43.392019 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:35:43.393571 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.393583 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.393585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.393797 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.394053 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:35:43.394054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:35:43.394054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:35:43.394164 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:35:43.394538 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:35:43.394542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:35:43.394657 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 09:35:43.394738 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:35:43.394852 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260926 09:35:43.395601 16753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260926 09:35:43.395709 16753 heartbeater.cc:464] Registering TS with master...
I20260926 09:35:43.395893 16753 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260926 09:35:43.396080 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.388594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.0.98.62:37461)
0926 09:35:43.388721 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.388723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.388865 (+ 142us) server_negotiation.cc:407] Connection header received
0926 09:35:43.388894 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.388895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.388907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.388983 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:43.389544 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.390114 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.391205 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.391544 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.391632 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.393141 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:43.393142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:43.393142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:43.393148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:43.394820 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.395215 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.395216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.395217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.395308 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.395540 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.395541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.395541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.395698 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:43.395858 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.395899 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.395980 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260926 09:35:43.396268 16289 ts_manager.cc:194] Registered new tserver with Master: 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329)
I20260926 09:35:43.396728 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34353
I20260926 09:35:43.396836 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:35:43.399308 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.397191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.0.98.62:37461)
0926 09:35:43.397256 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.397257 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.397312 (+ 55us) server_negotiation.cc:407] Connection header received
0926 09:35:43.397372 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.397373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.397401 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.397478 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:35:43.397543 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.397987 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.398742 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.398917 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.398964 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.399081 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:35:43.399115 (+ 34us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 09:35:43.399224 (+ 109us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53782: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 09:35:43.399384 16756 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53782: token expired
W20260926 09:35:43.399480 16223 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37461)
I20260926 09:35:43.404691 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.399752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.0.98.62:37461)
0926 09:35:43.399800 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.399802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.399850 (+ 48us) server_negotiation.cc:407] Connection header received
0926 09:35:43.399899 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.399900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.399906 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.399956 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:35:43.400358 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.400756 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.401350 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.401538 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.401583 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.402653 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:35:43.402654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:35:43.402654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:35:43.402659 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:35:43.403866 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.404162 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.404162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.404162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.404208 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:35:43.404358 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:35:43.404359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:35:43.404359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:35:43.404440 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:35:43.404507 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.404591 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.404642 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:35:43.405726 16289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:35:43.407277 16289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:35:43.409415 16552 tablet_service.cc:1467] Processing CreateTablet for tablet 6296982f79754067936271c49600567c (DEFAULT_TABLE table=test-table [id=c92d0bdb7e6049b596436926c17f5adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:43.409456 16412 tablet_service.cc:1467] Processing CreateTablet for tablet 6296982f79754067936271c49600567c (DEFAULT_TABLE table=test-table [id=c92d0bdb7e6049b596436926c17f5adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:43.409608 16552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6296982f79754067936271c49600567c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:43.409601 16412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6296982f79754067936271c49600567c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:43.411306 16509 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: Bootstrap starting.
I20260926 09:35:43.411304 16370 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Bootstrap starting.
I20260926 09:35:43.411790 16509 tablet_bootstrap.cc:654] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:43.411824 16370 tablet_bootstrap.cc:654] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:43.412387 16509 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: No bootstrap required, opened a new log
I20260926 09:35:43.412453 16509 ts_tablet_manager.cc:1397] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:43.412642 16370 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: No bootstrap required, opened a new log
I20260926 09:35:43.412684 16370 ts_tablet_manager.cc:1397] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:43.412744 16509 raft_consensus.cc:348] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.412839 16509 raft_consensus.cc:374] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:43.412894 16509 raft_consensus.cc:729] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93a8c5c4b8e46fb9d5ee08ebb623487, State: Initialized, Role: FOLLOWER
I20260926 09:35:43.412839 16370 raft_consensus.cc:348] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.412923 16370 raft_consensus.cc:374] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:43.412947 16509 consensus_queue.cc:260] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.412979 16370 raft_consensus.cc:729] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b9993cde4465a9419e4d5c05e9178, State: Initialized, Role: FOLLOWER
I20260926 09:35:43.413038 16509 ts_tablet_manager.cc:1428] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:35:43.413035 16370 consensus_queue.cc:260] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.413125 16370 ts_tablet_manager.cc:1428] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:35:43.413205 16619 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260926 09:35:43.414188 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.409201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43329 (local address 127.0.0.1:37332)
0926 09:35:43.409239 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.409250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:43.409288 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:43.409734 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:43.409735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:43.409737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:43.409776 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.409777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.410625 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.410626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.411507 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.411510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.413967 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.413968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.414033 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.414053 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 09:35:43.414112 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:35:43.414145 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 09:35:43.414211 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.409367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.0.98.3:43329)
0926 09:35:43.409539 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.409544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.409555 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:35:43.409589 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.409591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.409606 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.409728 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:43.409832 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.410539 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.412375 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.413891 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.413958 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.414018 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.414123 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.414153 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260926 09:35:43.414841 16687 tablet_service.cc:1467] Processing CreateTablet for tablet 6296982f79754067936271c49600567c (DEFAULT_TABLE table=test-table [id=c92d0bdb7e6049b596436926c17f5adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:35:43.415093 16687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6296982f79754067936271c49600567c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:35:43.417588 16763 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Bootstrap starting.
I20260926 09:35:43.418171 16763 tablet_bootstrap.cc:654] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Neither blocks nor log segments found. Creating new log.
I20260926 09:35:43.418673 16763 log.cc:824] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:35:43.419564 16763 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: No bootstrap required, opened a new log
I20260926 09:35:43.419633 16763 ts_tablet_manager.cc:1397] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:43.420966 16763 raft_consensus.cc:348] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.421105 16763 raft_consensus.cc:374] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:35:43.421128 16763 raft_consensus.cc:729] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ccf2da49cd941adb8feb6d1e6d3ec6a, State: Initialized, Role: FOLLOWER
I20260926 09:35:43.421239 16763 consensus_queue.cc:260] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.421460 16763 ts_tablet_manager.cc:1428] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:35:43.421516 16753 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260926 09:35:43.481720 16484 raft_consensus.cc:482] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:35:43.481861 16484 raft_consensus.cc:504] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.482189 16484 leader_election.cc:288] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329), c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
I20260926 09:35:43.486316 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.482506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53129 (local address 127.0.98.2:35309)
0926 09:35:43.482556 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.482558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.482662 (+ 104us) server_negotiation.cc:407] Connection header received
0926 09:35:43.482691 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.482692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.482737 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.482810 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:43.483024 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.483784 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.485299 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.485906 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.485967 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.486046 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.486176 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.486218 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:35:43.486300 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.482456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45719 (local address 127.0.98.3:43329)
0926 09:35:43.482506 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.482508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.482517 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:35:43.482599 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.482600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.482612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.482714 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:35:43.482933 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.483731 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.485299 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.485890 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.485967 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.485978 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.486176 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.486217 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:35:43.486326 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.482350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43329 (local address 127.0.98.1:45719)
0926 09:35:43.482441 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.482453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:43.482509 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:43.482730 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:43.482730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:43.482732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:43.482831 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.482833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.483810 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.483811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.485206 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.485209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.485974 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.485975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.486049 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.486079 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 09:35:43.486158 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:35:43.486205 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":120}
I20260926 09:35:43.486457 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.482409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35309 (local address 127.0.98.1:53129)
0926 09:35:43.482584 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.482593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:35:43.482646 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:35:43.482823 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:35:43.482824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:35:43.482825 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:35:43.482890 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.482891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.483852 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.483852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.485206 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:35:43.485209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.485990 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.485991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:35:43.486049 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.486080 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 09:35:43.486170 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:35:43.486205 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":38,"thread_start_us":102,"threads_started":1}
I20260926 09:35:43.486753 16707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6296982f79754067936271c49600567c" candidate_uuid: "df3b9993cde4465a9419e4d5c05e9178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" is_pre_election: true
I20260926 09:35:43.486932 16707 raft_consensus.cc:2454] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3b9993cde4465a9419e4d5c05e9178 in term 0.
I20260926 09:35:43.487020 16572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6296982f79754067936271c49600567c" candidate_uuid: "df3b9993cde4465a9419e4d5c05e9178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" is_pre_election: true
I20260926 09:35:43.487185 16572 raft_consensus.cc:2454] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3b9993cde4465a9419e4d5c05e9178 in term 0.
I20260926 09:35:43.487202 16366 leader_election.cc:302] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ccf2da49cd941adb8feb6d1e6d3ec6a, df3b9993cde4465a9419e4d5c05e9178; no voters:
I20260926 09:35:43.487360 16484 raft_consensus.cc:2781] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:35:43.487424 16484 raft_consensus.cc:482] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:35:43.487457 16484 raft_consensus.cc:3037] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:43.488281 16484 raft_consensus.cc:504] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.488425 16484 leader_election.cc:288] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 1 election: Requested vote from peers 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329), c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
I20260926 09:35:43.488574 16707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6296982f79754067936271c49600567c" candidate_uuid: "df3b9993cde4465a9419e4d5c05e9178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a"
W20260926 09:35:43.488590 16620 tablet.cc:2369] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:35:43.488600 16572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6296982f79754067936271c49600567c" candidate_uuid: "df3b9993cde4465a9419e4d5c05e9178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487"
I20260926 09:35:43.488691 16572 raft_consensus.cc:3037] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:43.488679 16707 raft_consensus.cc:3037] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:35:43.489352 16572 raft_consensus.cc:2454] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3b9993cde4465a9419e4d5c05e9178 in term 1.
I20260926 09:35:43.489357 16707 raft_consensus.cc:2454] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3b9993cde4465a9419e4d5c05e9178 in term 1.
I20260926 09:35:43.489544 16364 leader_election.cc:302] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c93a8c5c4b8e46fb9d5ee08ebb623487, df3b9993cde4465a9419e4d5c05e9178; no voters:
I20260926 09:35:43.489650 16484 raft_consensus.cc:2781] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:35:43.489790 16484 raft_consensus.cc:686] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 LEADER]: Becoming Leader. State: Replica: df3b9993cde4465a9419e4d5c05e9178, State: Running, Role: LEADER
I20260926 09:35:43.489878 16484 consensus_queue.cc:237] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:35:43.490643 16288 catalog_manager.cc:5495] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3b9993cde4465a9419e4d5c05e9178 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "df3b9993cde4465a9419e4d5c05e9178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } health_report { overall_health: HEALTHY } } }
I20260926 09:35:43.506245 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.503850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.0.98.1:43797)
0926 09:35:43.503892 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:35:43.503894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:35:43.503903 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:35:43.503969 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:35:43.503970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:35:43.503994 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:35:43.504071 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:35:43.504215 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.504710 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.505484 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:35:43.505696 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:35:43.505773 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:35:43.505934 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:35:43.506030 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:35:43.506113 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:35:43.506139 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 09:35:43.506163 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:35:43.507754 16572 raft_consensus.cc:1261] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Refusing update from remote peer df3b9993cde4465a9419e4d5c05e9178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:35:43.507797 16707 raft_consensus.cc:1261] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 FOLLOWER]: Refusing update from remote peer df3b9993cde4465a9419e4d5c05e9178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:35:43.508086 16767 consensus_queue.cc:1035] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:35:43.508240 16484 consensus_queue.cc:1035] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:35:43.510630 16771 mvcc.cc:205] Tried to move back new op lower bound from 7333541247005265920 to 7333541246935207936. Current Snapshot: MvccSnapshot[applied={T|T < 7333541247005265920}]
I20260926 09:35:43.510942 16773 mvcc.cc:205] Tried to move back new op lower bound from 7333541247005265920 to 7333541246935207936. Current Snapshot: MvccSnapshot[applied={T|T < 7333541247005265920}]
I20260926 09:35:43.512475 16287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:35:43.512552 16287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:35:43.513244 16287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Sending DeleteTablet for 3 replicas of tablet 6296982f79754067936271c49600567c
I20260926 09:35:43.513547 16552 tablet_service.cc:1514] Processing DeleteTablet for tablet 6296982f79754067936271c49600567c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48222
I20260926 09:35:43.513638 16624 tablet_replica.cc:330] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: stopping tablet replica
I20260926 09:35:43.513623 16412 tablet_service.cc:1514] Processing DeleteTablet for tablet 6296982f79754067936271c49600567c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34150
I20260926 09:35:43.513681 16624 raft_consensus.cc:2229] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:43.513669 16687 tablet_service.cc:1514] Processing DeleteTablet for tablet 6296982f79754067936271c49600567c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:35:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37332
I20260926 09:35:43.513721 16624 raft_consensus.cc:2258] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:43.513803 16487 tablet_replica.cc:330] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: stopping tablet replica
I20260926 09:35:43.513852 16487 raft_consensus.cc:2229] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:35:43.513932 16487 raft_consensus.cc:2258] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:43.514125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 16327
I20260926 09:35:43.514190 16487 ts_tablet_manager.cc:1905] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:43.514690 16779 tablet_replica.cc:330] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: stopping tablet replica
I20260926 09:35:43.514791 16779 raft_consensus.cc:2229] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:35:43.514914 16779 raft_consensus.cc:2258] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:35:43.515349 16779 ts_tablet_manager.cc:1905] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:43.517069 16624 ts_tablet_manager.cc:1905] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:35:43.517266 16779 ts_tablet_manager.cc:1918] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:43.517334 16779 log.cc:1196] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/6296982f79754067936271c49600567c
I20260926 09:35:43.517621 16779 ts_tablet_manager.cc:1939] T 6296982f79754067936271c49600567c P 9ccf2da49cd941adb8feb6d1e6d3ec6a: Deleting consensus metadata
I20260926 09:35:43.518002 16275 catalog_manager.cc:4841] TS 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329): tablet 6296982f79754067936271c49600567c (table test-table [id=c92d0bdb7e6049b596436926c17f5adc]) successfully deleted
I20260926 09:35:43.518463 16624 ts_tablet_manager.cc:1918] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:35:43.518510 16624 log.cc:1196] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/6296982f79754067936271c49600567c
I20260926 09:35:43.518656 16624 ts_tablet_manager.cc:1939] T 6296982f79754067936271c49600567c P c93a8c5c4b8e46fb9d5ee08ebb623487: Deleting consensus metadata
I20260926 09:35:43.519160 16273 catalog_manager.cc:4841] TS c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309): tablet 6296982f79754067936271c49600567c (table test-table [id=c92d0bdb7e6049b596436926c17f5adc]) successfully deleted
W20260926 09:35:43.522091 16275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43797 (error 108)
I20260926 09:35:43.522413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 16483
I20260926 09:35:43.522581 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:35:43.522436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43797 (local address 127.0.0.1:34166)
0926 09:35:43.522497 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:35:43.522542 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 09:35:43.523109 16275 catalog_manager.cc:4584] TS df3b9993cde4465a9419e4d5c05e9178 (127.0.98.1:43797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6296982f79754067936271c49600567c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43797: connect: Connection refused (error 111)
I20260926 09:35:43.527279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 16622
I20260926 09:35:43.531814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 16256
I20260926 09:36:03.536872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37461
--webserver_interface=127.0.98.62
--webserver_port=44655
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:03.619187 16781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:03.619410 16781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:03.619433 16781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:03.620519 16781 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:36:03.620555 16781 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:36:03.620568 16781 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:36:03.620580 16781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:03.620594 16781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:03.620606 16781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:03.620621 16781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:03.620635 16781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:03.622046 16781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:03.622283 16781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:03.622457 16781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:03.627436 16789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:03.627475 16781 server_base.cc:1034] running on GCE node
W20260926 09:36:03.627508 16787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:03.627436 16786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:03.627826 16781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:03.628121 16781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:03.629259 16781 hybrid_clock.cc:648] HybridClock initialized: now 1790415363629250 us; error 29 us; skew 500 ppm
Sep 26 09:36:03 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:03.631680 16781 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:03.632144 16781 webserver.cc:460] Webserver started at http://127.0.98.62:44655/ using document root <none> and password file <none>
I20260926 09:36:03.632378 16781 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:03.632423 16781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:03.633713 16781 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:03.634347 16796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:03.634523 16781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:03.634622 16781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "382bd0779d1942b799bf8921c3db8275"
format_stamp: "Formatted at 2026-09-26 09:35:21 on dist-test-slave-21h8"
server_key: "870bd2580e0c42bf7e98e1cae75534c2"
server_key_iv: "3c6163a8a6dcdc454432096c15855e00"
server_key_version: "encryptionkey@0"
I20260926 09:36:03.634951 16781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:03.648597 16781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:03.650811 16781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:03.651000 16781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:03.655301 16781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37461
I20260926 09:36:03.655406 16848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37461 every 8 connection(s)
I20260926 09:36:03.655800 16781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:03.657227 16849 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:36:03.657810 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Bootstrap starting.
I20260926 09:36:03.661868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 16781
I20260926 09:36:03.662125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43797
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=39801
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:36:03.666347 16849 log.cc:824] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:03.668569 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:36:03.668895 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: Bootstrap complete.
I20260926 09:36:03.671087 16849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:36:03.671357 16849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382bd0779d1942b799bf8921c3db8275, State: Initialized, Role: FOLLOWER
I20260926 09:36:03.671553 16849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:36:03.671618 16849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:03.671676 16849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:03.671725 16849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 2 FOLLOWER]: Advancing to term 3
I20260926 09:36:03.672717 16849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:36:03.672803 16849 leader_election.cc:302] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382bd0779d1942b799bf8921c3db8275; no voters:
I20260926 09:36:03.672988 16849 leader_election.cc:288] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 09:36:03.673046 16853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 09:36:03.673254 16849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:03.673238 16853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [term 3 LEADER]: Becoming Leader. State: Replica: 382bd0779d1942b799bf8921c3db8275, State: Running, Role: LEADER
I20260926 09:36:03.673460 16853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260926 09:36:03.673820 16855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382bd0779d1942b799bf8921c3db8275. Latest consensus state: current_term: 3 leader_uuid: "382bd0779d1942b799bf8921c3db8275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } }
I20260926 09:36:03.673892 16855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:03.674082 16854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "382bd0779d1942b799bf8921c3db8275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd0779d1942b799bf8921c3db8275" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } }
I20260926 09:36:03.674190 16854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:03.675137 16869 catalog_manager.cc:1259] Loaded cluster ID: e266de89e3db48be94e94234821d0321
I20260926 09:36:03.675194 16869 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: loading cluster ID for follower catalog manager: success
I20260926 09:36:03.676388 16869 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 382bd0779d1942b799bf8921c3db8275: acquiring CA information for follower catalog manager: success
F20260926 09:36:03.676764 16869 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdc9cfb89ad google::LogMessage::Fail() at ??:0
@ 0x7fdc9cfbb88a google::LogMessage::SendToLog() at ??:0
@ 0x7fdc9cfb84de google::LogMessage::Flush() at ??:0
@ 0x7fdc9cfb9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdc9f725104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdca4f116b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fdca4f11989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fdca4f11e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fdca4f148c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fdc9ee7b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdc9fcdf6db start_thread at ??:0
@ 0x7fdc9bab661f clone at ??:0
W20260926 09:36:03.762751 16851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:03.762951 16851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:03.762975 16851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:03.764168 16851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:03.764214 16851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:03.764241 16851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:36:03.765642 16851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39801
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:03.765877 16851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:03.766122 16851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:03.771054 16887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:03.771068 16888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:03.771139 16851 server_base.cc:1034] running on GCE node
W20260926 09:36:03.771344 16890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:03.771548 16851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:03.771826 16851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:03.773005 16851 hybrid_clock.cc:648] HybridClock initialized: now 1790415363772993 us; error 27 us; skew 500 ppm
Sep 26 09:36:03 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:03.775568 16851 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:36:03.776053 16851 webserver.cc:460] Webserver started at http://127.0.98.1:39801/ using document root <none> and password file <none>
I20260926 09:36:03.776297 16851 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:03.776366 16851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:03.777781 16851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:03.778627 16897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:03.778846 16851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:03.778955 16851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "df3b9993cde4465a9419e4d5c05e9178"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "fa867719a395a752d37d43f29c544490"
server_key_iv: "7fc9b2c1f3999282c7a5a90fea768758"
server_key_version: "encryptionkey@0"
I20260926 09:36:03.779439 16851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:03.809481 16851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:03.812131 16851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:03.812286 16851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:03.812551 16851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:03.813109 16904 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:36:03.814616 16851 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:36:03.814680 16851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:36:03.814716 16851 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:36:03.815626 16851 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:36:03.815675 16851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:36:03.815768 16904 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Bootstrap starting.
I20260926 09:36:03.817427 16904 log.cc:824] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:03.820425 16904 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:36:03.820729 16904 tablet_bootstrap.cc:492] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Bootstrap complete.
I20260926 09:36:03.820878 16904 ts_tablet_manager.cc:1397] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 09:36:03.822736 16904 raft_consensus.cc:348] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:36:03.823065 16904 raft_consensus.cc:729] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b9993cde4465a9419e4d5c05e9178, State: Initialized, Role: FOLLOWER
I20260926 09:36:03.823136 16851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43797
I20260926 09:36:03.823271 16904 consensus_queue.cc:260] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:36:03.823529 17012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43797 every 8 connection(s)
I20260926 09:36:03.823606 16904 ts_tablet_manager.cc:1428] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:03.823539 16851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:36:03.827625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 16851
I20260926 09:36:03.827908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35309
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=39975
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:03.877498 17016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:03.824382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:57919)
0926 09:36:03.824546 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:03.824577 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:03.824704 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:03.877284 (+ 52580us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
W20260926 09:36:03.877717 17014 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37461 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37461: Client connection negotiation failed: client connection to 127.0.98.62:37461: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:36:03.878087 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:03.877956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:42847)
0926 09:36:03.878014 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:03.878056 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:36:03.878476 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:03.878387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:46729)
0926 09:36:03.878438 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:03.878461 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 09:36:03.878605 17014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:36:03.912381 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:03.912580 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:03.912603 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:03.913765 17017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:03.913808 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:03.913861 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:36:03.915378 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39975
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:03.915606 17017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:03.915836 17017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:36:03.920615 17017 server_base.cc:1034] running on GCE node
W20260926 09:36:03.920660 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:03.920761 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:03.920625 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:03.921084 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:03.921330 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:03.922508 17017 hybrid_clock.cc:648] HybridClock initialized: now 1790415363922498 us; error 28 us; skew 500 ppm
Sep 26 09:36:03 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:03.924969 17017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:36:03.925439 17017 webserver.cc:460] Webserver started at http://127.0.98.2:39975/ using document root <none> and password file <none>
I20260926 09:36:03.925664 17017 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:03.925733 17017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:03.927035 17017 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:03.927635 17032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:03.927815 17017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:03.927927 17017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "0177fc1894daf74f744beccd4909b86c"
server_key_iv: "d280a76f7ab9a8157c5265eeaa064e59"
server_key_version: "encryptionkey@0"
I20260926 09:36:03.928287 17017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:03.943964 17017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:03.946376 17017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:03.946538 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:03.946775 17017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:03.947082 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:03.947116 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:03.947158 17017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:03.947194 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:03.953263 17017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35309
I20260926 09:36:03.953312 17145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35309 every 8 connection(s)
I20260926 09:36:03.953640 17017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:36:03.954615 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:03.954256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:47411)
0926 09:36:03.954407 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:03.954477 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260926 09:36:03.954990 17146 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37461 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37461: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260926 09:36:03.955364 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:03.955211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:44905)
0926 09:36:03.955268 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:03.955298 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 09:36:03.955647 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:03.955560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:43063)
0926 09:36:03.955602 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:03.955625 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 09:36:03.955744 17146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:36:03.963178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 17017
I20260926 09:36:03.963485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43329
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=37039
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:04.043869 17149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:04.044054 17149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:04.044076 17149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:04.045121 17149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:04.045156 17149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:04.045181 17149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:36:04.046504 17149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37039
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:04.046679 17149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:04.046834 17149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:04.051733 17155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:04.051745 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:04.051745 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:04.051977 17149 server_base.cc:1034] running on GCE node
I20260926 09:36:04.052168 17149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:04.052364 17149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:04.053637 17149 hybrid_clock.cc:648] HybridClock initialized: now 1790415364053624 us; error 26 us; skew 500 ppm
Sep 26 09:36:04 dist-test-slave-21h8 krb5kdc[15713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:04.056130 17149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:36:04.056568 17149 webserver.cc:460] Webserver started at http://127.0.98.3:37039/ using document root <none> and password file <none>
I20260926 09:36:04.056754 17149 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:04.056793 17149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:04.057996 17149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:36:04.058600 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:04.058804 17149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:36:04.058878 17149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a"
format_stamp: "Formatted at 2026-09-26 09:35:22 on dist-test-slave-21h8"
server_key: "b969be9f7436655b51d0d0cf8e423eba"
server_key_iv: "479578228457755c33f9579175c1bd5b"
server_key_version: "encryptionkey@0"
I20260926 09:36:04.059190 17149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:04.076268 17149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:04.078522 17149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:04.078639 17149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:04.078832 17149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:04.079140 17149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:04.079185 17149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:04.079228 17149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:04.079255 17149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:04.085057 17149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43329
I20260926 09:36:04.085140 17277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43329 every 8 connection(s)
I20260926 09:36:04.085454 17149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:36:04.086457 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:04.086101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:46249)
0926 09:36:04.086259 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:04.086329 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
W20260926 09:36:04.086858 17278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37461 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37461: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260926 09:36:04.087143 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:04.087050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:42485)
0926 09:36:04.087096 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:04.087120 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 09:36:04.087478 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:04.087397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:60857)
0926 09:36:04.087437 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:04.087459 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 09:36:04.087585 17278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:36:04.088248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 17149
I20260926 09:36:04.879191 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:04.878968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:33837)
0926 09:36:04.879098 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:04.879130 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260926 09:36:04.956264 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:04.956075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:54443)
0926 09:36:04.956189 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:04.956221 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260926 09:36:05.088056 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:05.087864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:59037)
0926 09:36:05.087979 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:05.088013 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260926 09:36:05.363080 17285 raft_consensus.cc:482] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:05.363201 17285 raft_consensus.cc:504] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:36:05.363620 17285 leader_election.cc:288] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329), c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
Sep 26 09:36:05 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 09:36:05 dist-test-slave-21h8 krb5kdc[15713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 09:36:05.374449 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:05.363810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35959 (local address 127.0.98.3:43329)
0926 09:36:05.363991 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:05.364004 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:05.364043 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:36:05.364534 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:05.364536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:05.364594 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:05.364741 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:05.365547 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:05.366651 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:05.367719 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:05.368127 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:05.368254 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:05.370862 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:05.370866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:05.370866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:05.370893 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:05.372666 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:05.373268 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:05.373271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:05.373271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:05.373345 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:05.373542 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:05.373543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:05.373543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:05.373800 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:05.373872 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:05.374214 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 09:36:05.374336 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260926 09:36:05.374449 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:05.363906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42465 (local address 127.0.98.2:35309)
0926 09:36:05.364070 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:05.364082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:05.364113 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:36:05.364553 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:05.364556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:05.364594 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:05.364735 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:05.365761 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:05.366730 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:05.367742 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:05.368129 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:05.368239 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:05.370570 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:05.370575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:05.370575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:05.370587 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:05.372657 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:05.373056 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:05.373058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:05.373059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:05.373142 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:05.373410 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:05.373412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:05.373412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:05.373745 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:05.373833 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:05.374205 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 09:36:05.374336 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":52,"threads_started":1}
I20260926 09:36:05.374441 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:05.363736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43329 (local address 127.0.98.1:35959)
0926 09:36:05.363974 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:05.363986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:05.364046 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:05.364768 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:05.364771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:05.365194 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:05.365415 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:05.365421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:05.366744 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:05.366745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:05.367634 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:05.367637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:05.368230 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:05.368231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:05.368345 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:05.368929 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:05.368937 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:05.370770 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:05.373051 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:05.373052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:05.373052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:05.373197 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:05.373438 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:05.373438 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:05.373438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:05.373485 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:05.373864 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:05.373865 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:05.374005 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 09:36:05.374123 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 09:36:05.374302 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":79}
I20260926 09:36:05.374454 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:05.363804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35309 (local address 127.0.98.1:42465)
0926 09:36:05.363927 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:05.363940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:05.364002 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:05.364768 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:05.364771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:05.365193 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:05.365627 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:05.365630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:05.366828 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:05.366828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:05.367634 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:05.367636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:05.368217 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:05.368218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:05.368349 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:05.368990 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:05.368994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:05.370463 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:05.372738 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:05.372744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:05.372746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:05.372982 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:05.373221 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:05.373222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:05.373223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:05.373341 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:05.373816 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:05.373821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:05.374005 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 09:36:05.374123 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 09:36:05.374355 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":134,"thread_start_us":70,"threads_started":1}
W20260926 09:36:05.375351 16900 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329): Not found: Tablet not found: 6296982f79754067936271c49600567c
W20260926 09:36:05.375734 16898 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309): Not found: Tablet not found: 6296982f79754067936271c49600567c
I20260926 09:36:05.375782 16898 leader_election.cc:302] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3b9993cde4465a9419e4d5c05e9178; no voters: 9ccf2da49cd941adb8feb6d1e6d3ec6a, c93a8c5c4b8e46fb9d5ee08ebb623487
I20260926 09:36:05.375947 17285 raft_consensus.cc:2726] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:36:05.879827 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:05.879641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:36465)
0926 09:36:05.879748 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:05.879782 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260926 09:36:05.956887 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:05.956691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:48829)
0926 09:36:05.956804 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:05.956837 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260926 09:36:06.088665 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:06.088473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:42565)
0926 09:36:06.088591 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:06.088628 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260926 09:36:06.880509 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:06.880275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:44059)
0926 09:36:06.880407 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:06.880440 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260926 09:36:06.957576 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:06.957350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:40793)
0926 09:36:06.957491 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:06.957539 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260926 09:36:07.089300 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:07.089094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:36703)
0926 09:36:07.089228 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:07.089263 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260926 09:36:07.170308 17295 raft_consensus.cc:482] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:07.170429 17295 raft_consensus.cc:504] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:36:07.170640 17295 leader_election.cc:288] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329), c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
W20260926 09:36:07.171120 16900 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329): Not found: Tablet not found: 6296982f79754067936271c49600567c
W20260926 09:36:07.171279 16898 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309): Not found: Tablet not found: 6296982f79754067936271c49600567c
I20260926 09:36:07.171377 16898 leader_election.cc:302] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3b9993cde4465a9419e4d5c05e9178; no voters: 9ccf2da49cd941adb8feb6d1e6d3ec6a, c93a8c5c4b8e46fb9d5ee08ebb623487
I20260926 09:36:07.171468 17295 raft_consensus.cc:2726] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:36:07.881263 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:07.881020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:47645)
0926 09:36:07.881158 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:07.881194 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260926 09:36:07.958227 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:07.958029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:47765)
0926 09:36:07.958155 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:07.958189 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260926 09:36:08.089902 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:08.089724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:56775)
0926 09:36:08.089831 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:08.089870 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260926 09:36:08.882033 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:08.881793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:46141)
0926 09:36:08.881944 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:08.881985 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260926 09:36:08.958899 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:08.958674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:54223)
0926 09:36:08.958810 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:08.958861 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260926 09:36:09.090764 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:09.090494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:57335)
0926 09:36:09.090689 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:09.090727 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":57,"threads_started":1}
I20260926 09:36:09.840022 17302 raft_consensus.cc:482] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:09.840159 17302 raft_consensus.cc:504] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:36:09.840359 17302 leader_election.cc:288] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329), c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
W20260926 09:36:09.840739 16900 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329): Not found: Tablet not found: 6296982f79754067936271c49600567c
W20260926 09:36:09.840833 16898 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309): Not found: Tablet not found: 6296982f79754067936271c49600567c
I20260926 09:36:09.840929 16898 leader_election.cc:302] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3b9993cde4465a9419e4d5c05e9178; no voters: 9ccf2da49cd941adb8feb6d1e6d3ec6a, c93a8c5c4b8e46fb9d5ee08ebb623487
I20260926 09:36:09.841030 17302 raft_consensus.cc:2726] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:36:09.882753 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:09.882555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:55945)
0926 09:36:09.882679 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:09.882708 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260926 09:36:09.959547 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:09.959361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:41387)
0926 09:36:09.959474 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:09.959507 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260926 09:36:10.091488 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:10.091251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:44493)
0926 09:36:10.091416 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:10.091452 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":169,"threads_started":1}
I20260926 09:36:10.883531 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:10.883266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:59573)
0926 09:36:10.883449 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:10.883485 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260926 09:36:10.960208 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:10.959969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:58447)
0926 09:36:10.960122 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:10.960166 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260926 09:36:11.092213 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:11.092023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:48157)
0926 09:36:11.092136 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:11.092171 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260926 09:36:11.884240 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:11.884015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:53983)
0926 09:36:11.884142 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:11.884191 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260926 09:36:11.960913 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:11.960700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:51751)
0926 09:36:11.960829 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:11.960871 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260926 09:36:12.092909 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:12.092680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:44173)
0926 09:36:12.092821 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:12.092863 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260926 09:36:12.884907 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:12.884714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:46289)
0926 09:36:12.884829 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:12.884861 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260926 09:36:12.961699 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:12.961458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:56373)
0926 09:36:12.961603 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:12.961653 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260926 09:36:13.093566 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:13.093369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:55959)
0926 09:36:13.093488 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:13.093523 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260926 09:36:13.885524 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:13.885327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:41111)
0926 09:36:13.885443 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:13.885479 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260926 09:36:13.962396 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:13.962151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:52839)
0926 09:36:13.962318 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:13.962354 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":57,"threads_started":1}
I20260926 09:36:14.058218 17317 raft_consensus.cc:482] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:14.058314 17317 raft_consensus.cc:504] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:36:14.058529 17317 leader_election.cc:288] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329), c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
W20260926 09:36:14.058921 16900 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329): Not found: Tablet not found: 6296982f79754067936271c49600567c
W20260926 09:36:14.058995 16898 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309): Not found: Tablet not found: 6296982f79754067936271c49600567c
I20260926 09:36:14.059070 16898 leader_election.cc:302] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3b9993cde4465a9419e4d5c05e9178; no voters: 9ccf2da49cd941adb8feb6d1e6d3ec6a, c93a8c5c4b8e46fb9d5ee08ebb623487
I20260926 09:36:14.059158 17317 raft_consensus.cc:2726] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:36:14.094419 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:14.094176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:35113)
0926 09:36:14.094331 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:14.094367 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260926 09:36:14.886250 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:14.886030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:48297)
0926 09:36:14.886162 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:14.886203 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260926 09:36:14.962985 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:14.962806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:59627)
0926 09:36:14.962914 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:14.962946 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260926 09:36:15.095178 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:15.094975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:48967)
0926 09:36:15.095100 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:15.095136 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260926 09:36:15.886926 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:15.886706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:60817)
0926 09:36:15.886841 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:15.886880 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260926 09:36:15.905117 17323 raft_consensus.cc:482] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:15.905205 17323 raft_consensus.cc:504] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ccf2da49cd941adb8feb6d1e6d3ec6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43329 } } peers { permanent_uuid: "c93a8c5c4b8e46fb9d5ee08ebb623487" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35309 } } peers { permanent_uuid: "df3b9993cde4465a9419e4d5c05e9178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43797 } }
I20260926 09:36:15.905362 17323 leader_election.cc:288] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329), c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309)
W20260926 09:36:15.905745 16900 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ccf2da49cd941adb8feb6d1e6d3ec6a (127.0.98.3:43329): Not found: Tablet not found: 6296982f79754067936271c49600567c
W20260926 09:36:15.905848 16898 leader_election.cc:341] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c93a8c5c4b8e46fb9d5ee08ebb623487 (127.0.98.2:35309): Not found: Tablet not found: 6296982f79754067936271c49600567c
I20260926 09:36:15.905879 16900 leader_election.cc:302] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3b9993cde4465a9419e4d5c05e9178; no voters: 9ccf2da49cd941adb8feb6d1e6d3ec6a, c93a8c5c4b8e46fb9d5ee08ebb623487
I20260926 09:36:15.905947 17323 raft_consensus.cc:2726] T 6296982f79754067936271c49600567c P df3b9993cde4465a9419e4d5c05e9178 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:36:15.963713 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:15.963509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:39941)
0926 09:36:15.963637 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:15.963672 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260926 09:36:16.095903 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:16.095692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:53469)
0926 09:36:16.095834 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:16.095866 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260926 09:36:16.887626 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:16.887405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:53427)
0926 09:36:16.887548 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:16.887582 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260926 09:36:16.964388 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:16.964189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:47293)
0926 09:36:16.964310 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:16.964344 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260926 09:36:17.096544 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:17.096316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:35785)
0926 09:36:17.096462 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:17.096495 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260926 09:36:17.888249 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:17.888048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:46689)
0926 09:36:17.888177 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:17.888205 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260926 09:36:17.965112 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:17.964870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:53405)
0926 09:36:17.965022 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:17.965067 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260926 09:36:18.097144 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:18.096948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:58891)
0926 09:36:18.097063 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:18.097100 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260926 09:36:18.888969 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:18.888752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:59451)
0926 09:36:18.888888 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:18.888926 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":55,"threads_started":1}
I20260926 09:36:18.965838 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:18.965615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:54843)
0926 09:36:18.965763 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:18.965797 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260926 09:36:19.097842 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:19.097628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:37085)
0926 09:36:19.097751 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:19.097796 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260926 09:36:19.098855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 16851
I20260926 09:36:19.103794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 17017
I20260926 09:36:19.107847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 17149
2026-09-26T09:36:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58371 ms)
[----------] 30 tests from SecurityITest (233003 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:19 dist-test-slave-21h8 krb5kdc[17337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:19 dist-test-slave-21h8 krb5kdc[17337](info): set up 2 sockets
Sep 26 09:36:19 dist-test-slave-21h8 krb5kdc[17337](info): commencing operation
krb5kdc: starting...
W20260926 09:36:20.150939 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:20.264398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:20.345731 17350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:20.345968 17350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:20.346004 17350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:20.347157 17350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:20.347225 17350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:20.347246 17350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:20.347266 17350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:20.347298 17350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:20.348737 17350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:20.349005 17350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:20.349239 17350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:20.354449 17355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:20.354643 17350 server_base.cc:1034] running on GCE node
W20260926 09:36:20.354449 17356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:20.354486 17358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:20.354969 17350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:20.355264 17350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:20.356432 17350 hybrid_clock.cc:648] HybridClock initialized: now 1790415380356420 us; error 28 us; skew 500 ppm
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:20.358850 17350 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:20.359195 17350 webserver.cc:460] Webserver started at http://127.0.98.62:39935/ using document root <none> and password file <none>
I20260926 09:36:20.359434 17350 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:20.359489 17350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:20.359592 17350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:20.360551 17350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "6c22dab149e6491d889473745a527287"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "38ff2cb484f6fa6310d7c6fa094eb84a"
server_key_iv: "f3eb79f9d97a396bccb019bccfbc7d1b"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.360829 17350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c22dab149e6491d889473745a527287"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "38ff2cb484f6fa6310d7c6fa094eb84a"
server_key_iv: "f3eb79f9d97a396bccb019bccfbc7d1b"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.362064 17350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:36:20.362696 17365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.362850 17350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:20.362936 17350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "6c22dab149e6491d889473745a527287"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "38ff2cb484f6fa6310d7c6fa094eb84a"
server_key_iv: "f3eb79f9d97a396bccb019bccfbc7d1b"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.362993 17350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:20.386183 17350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:20.388013 17350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:20.388126 17350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:20.391929 17350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41775
I20260926 09:36:20.391992 17417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41775 every 8 connection(s)
I20260926 09:36:20.392323 17350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:20.393189 17418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:20.395437 17418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287: Bootstrap starting.
I20260926 09:36:20.395985 17418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:20.396266 17418 log.cc:824] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:20.396943 17418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287: No bootstrap required, opened a new log
I20260926 09:36:20.398451 17418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c22dab149e6491d889473745a527287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } }
I20260926 09:36:20.398669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 17350
I20260926 09:36:20.398694 17418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:20.398736 17418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c22dab149e6491d889473745a527287, State: Initialized, Role: FOLLOWER
I20260926 09:36:20.398766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:20.398893 17418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c22dab149e6491d889473745a527287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } }
I20260926 09:36:20.398962 392 external_mini_cluster.cc:1428] Setting key 12d5069eaedcd0493afdecd023649260
I20260926 09:36:20.398975 17418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:20.399009 17418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:20.399061 17418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:20.400141 17418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c22dab149e6491d889473745a527287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } }
I20260926 09:36:20.400259 17418 leader_election.cc:302] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c22dab149e6491d889473745a527287; no voters:
I20260926 09:36:20.400424 17418 leader_election.cc:288] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:20.400548 17423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:20.400733 17418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:20.400748 17423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [term 1 LEADER]: Becoming Leader. State: Replica: 6c22dab149e6491d889473745a527287, State: Running, Role: LEADER
I20260926 09:36:20.400957 17423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c22dab149e6491d889473745a527287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } }
I20260926 09:36:20.401540 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c22dab149e6491d889473745a527287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c22dab149e6491d889473745a527287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } } }
I20260926 09:36:20.401645 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:20.402148 17432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:20.402320 17425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c22dab149e6491d889473745a527287. Latest consensus state: current_term: 1 leader_uuid: "6c22dab149e6491d889473745a527287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c22dab149e6491d889473745a527287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } } }
I20260926 09:36:20.402417 17425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:20.402613 17432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:20.404280 17432 catalog_manager.cc:1347] Generated new cluster ID: ed4950299e164b3e87e21f5645953059
I20260926 09:36:20.404340 17432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:20.410501 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.399440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.0.98.62:41775)
0926 09:36:20.399641 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:20.399656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:20.399682 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:36:20.399770 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:20.399775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:20.399814 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:20.399921 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:20.400782 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.402605 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.403731 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.404316 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.404423 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.406986 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:20.406991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:20.406991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:20.407005 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:20.408862 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.409272 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.409275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.409275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.409385 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.409620 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.409621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.409621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.409861 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:20.410071 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:20.410140 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 09:36:20.410293 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":126,"threads_started":1}
I20260926 09:36:20.426371 17432 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:20.426934 17432 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:20.430119 17432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287: Generated new TSK 0
I20260926 09:36:20.430330 17432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:36:20.461578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:20.564545 17444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:20.564761 17444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:20.564783 17444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:20.566156 17444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:20.566195 17444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:20.566221 17444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:36:20.567961 17444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:20.568233 17444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:20.568451 17444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:20.573364 17449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:20.573370 17450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:20.573395 17444 server_base.cc:1034] running on GCE node
W20260926 09:36:20.573374 17452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:20.573765 17444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:20.574019 17444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:20.575181 17444 hybrid_clock.cc:648] HybridClock initialized: now 1790415380575168 us; error 42 us; skew 500 ppm
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:20.577626 17444 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:36:20.578028 17444 webserver.cc:460] Webserver started at http://127.0.98.1:39221/ using document root <none> and password file <none>
I20260926 09:36:20.578254 17444 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:20.578321 17444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:20.578475 17444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:20.579576 17444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "174bc08df2724fb8a2fd1d5e2e1316a1"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "5328871c23bfaa14f7c5877336b71f17"
server_key_iv: "4b3b3c8286d4addf9dbba78a8ed2fc08"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.579926 17444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "174bc08df2724fb8a2fd1d5e2e1316a1"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "5328871c23bfaa14f7c5877336b71f17"
server_key_iv: "4b3b3c8286d4addf9dbba78a8ed2fc08"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.581175 17444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:36:20.581895 17459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.582067 17444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:20.582162 17444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "174bc08df2724fb8a2fd1d5e2e1316a1"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "5328871c23bfaa14f7c5877336b71f17"
server_key_iv: "4b3b3c8286d4addf9dbba78a8ed2fc08"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.582247 17444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:20.593315 17444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:20.594959 17444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:20.595111 17444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:20.595363 17444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:20.595671 17444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:20.595726 17444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.595758 17444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:20.595784 17444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.601547 17444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38433
I20260926 09:36:20.601590 17572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38433 every 8 connection(s)
I20260926 09:36:20.601958 17444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:36:20.606524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 17444
I20260926 09:36:20.606637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:36:20.606817 392 external_mini_cluster.cc:1428] Setting key 7902ad360995803eddefad591c9d353d
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:20.611928 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.602585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41775 (local address 127.0.98.1:40303)
0926 09:36:20.602735 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:20.602774 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:20.603760 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:20.604009 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:20.604012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:20.604245 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:20.604421 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.604424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.605090 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.605090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.605868 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.605869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.606203 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.606204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.606300 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.606825 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:20.606840 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:20.608661 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:20.610458 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:20.610464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:20.610466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:20.610680 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:20.610919 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:20.610920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:20.610920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:20.611015 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:20.611449 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:20.611456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:20.611583 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 09:36:20.611672 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:36:20.611782 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260926 09:36:20.611959 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.602606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40303 (local address 127.0.98.62:41775)
0926 09:36:20.602684 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:20.602685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:20.602860 (+ 175us) server_negotiation.cc:407] Connection header received
0926 09:36:20.603859 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:20.603877 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:20.603890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:20.603945 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:20.604497 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.604990 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.605943 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.606131 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.606181 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.608746 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:20.608747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:20.608747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:20.608753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:20.610373 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.610752 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.610753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.610753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.610815 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.611083 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.611084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.611084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.611234 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:20.611338 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:20.611725 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 09:36:20.611816 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:36:20.612442 17573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41775
I20260926 09:36:20.612543 17573 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:20.612751 17573 heartbeater.cc:511] Master 127.0.98.62:41775 requested a full tablet report, sending...
I20260926 09:36:20.613253 17382 ts_manager.cc:194] Registered new tserver with Master: 174bc08df2724fb8a2fd1d5e2e1316a1 (127.0.98.1:38433)
I20260926 09:36:20.613837 17382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40303
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:36:20.632047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:20.711592 17578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:20.711781 17578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:20.711812 17578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:20.712983 17578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:20.713027 17578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:20.713053 17578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:36:20.714515 17578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:20.714751 17578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:20.714982 17578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:20.719833 17586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:20.719851 17584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:20.719843 17583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:20.720104 17578 server_base.cc:1034] running on GCE node
I20260926 09:36:20.720296 17578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:20.720490 17578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:20.721623 17578 hybrid_clock.cc:648] HybridClock initialized: now 1790415380721610 us; error 29 us; skew 500 ppm
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:20.724007 17578 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:36:20.724447 17578 webserver.cc:460] Webserver started at http://127.0.98.2:44527/ using document root <none> and password file <none>
I20260926 09:36:20.724632 17578 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:20.724680 17578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:20.724766 17578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:20.725716 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "fbbf60093076461dac24a117ef33b6d4"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "2e0e0b52d1ee10206a235a8db3732a6e"
server_key_iv: "356a14dfe51b6dabaab945b8f8c5d0ae"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.725994 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fbbf60093076461dac24a117ef33b6d4"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "2e0e0b52d1ee10206a235a8db3732a6e"
server_key_iv: "356a14dfe51b6dabaab945b8f8c5d0ae"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.727157 17578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:36:20.727911 17593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.728084 17578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:20.728168 17578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "fbbf60093076461dac24a117ef33b6d4"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "2e0e0b52d1ee10206a235a8db3732a6e"
server_key_iv: "356a14dfe51b6dabaab945b8f8c5d0ae"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.728219 17578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:20.765169 17578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:20.767146 17578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:20.767261 17578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:20.767504 17578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:20.767786 17578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:20.767818 17578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.767840 17578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:20.767853 17578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.773716 17578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45043
I20260926 09:36:20.773787 17706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45043 every 8 connection(s)
I20260926 09:36:20.774086 17578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:36:20.776513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 17578
I20260926 09:36:20.776624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:36:20.776830 392 external_mini_cluster.cc:1428] Setting key 04242178fbc43a0a400970a799590044
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:20.784711 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.774781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59939 (local address 127.0.98.62:41775)
0926 09:36:20.774861 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:20.774864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:20.775022 (+ 158us) server_negotiation.cc:407] Connection header received
0926 09:36:20.776022 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:20.776023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:20.776033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:20.776085 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:20.776930 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.777553 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.779105 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.779466 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.779542 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.781687 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:20.781689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:20.781689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:20.781698 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:20.783366 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.783746 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.783747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.783747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.783814 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.784047 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.784049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.784049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.784199 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:20.784275 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:20.784506 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 09:36:20.784594 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:36:20.784976 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.774735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41775 (local address 127.0.98.2:59939)
0926 09:36:20.774907 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:20.774942 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:20.775932 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:20.776148 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:20.776152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:20.776515 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:20.776831 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.776837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.777625 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.777627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.779014 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.779018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.779547 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.779548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.779653 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.780151 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:20.780159 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:20.781592 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:20.783455 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:20.783461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:20.783463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:20.783672 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:20.783886 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:20.783886 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:20.783886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:20.783981 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:20.784267 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:20.784271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:20.784378 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 09:36:20.784459 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:36:20.784645 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260926 09:36:20.785406 17707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41775
I20260926 09:36:20.785497 17707 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:36:20.785694 17707 heartbeater.cc:511] Master 127.0.98.62:41775 requested a full tablet report, sending...
I20260926 09:36:20.786149 17382 ts_manager.cc:194] Registered new tserver with Master: fbbf60093076461dac24a117ef33b6d4 (127.0.98.2:45043)
I20260926 09:36:20.786530 17382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:36:20.802366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:20.883929 17712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:20.884127 17712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:20.884152 17712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:20.885303 17712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:20.885346 17712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:20.885375 17712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:36:20.886859 17712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:20.887035 17712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:20.887207 17712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:20.892192 17720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:20.892257 17712 server_base.cc:1034] running on GCE node
W20260926 09:36:20.892230 17717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:20.892221 17718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:20.892599 17712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:20.892844 17712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:20.894003 17712 hybrid_clock.cc:648] HybridClock initialized: now 1790415380893984 us; error 36 us; skew 500 ppm
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:20.896489 17712 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:36:20.896993 17712 webserver.cc:460] Webserver started at http://127.0.98.3:33719/ using document root <none> and password file <none>
I20260926 09:36:20.897203 17712 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:20.897258 17712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:20.897464 17712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:20.898486 17712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a451b7941d3449c876ffb1e2fcf379c"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "19804b11438e182ed8d5f4b5a6ff503a"
server_key_iv: "1617cb7eef50aa183f1d1e763afb5598"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.898833 17712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a451b7941d3449c876ffb1e2fcf379c"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "19804b11438e182ed8d5f4b5a6ff503a"
server_key_iv: "1617cb7eef50aa183f1d1e763afb5598"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.900179 17712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:20.900976 17727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.901165 17712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:20.901230 17712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "0a451b7941d3449c876ffb1e2fcf379c"
format_stamp: "Formatted at 2026-09-26 09:36:20 on dist-test-slave-21h8"
server_key: "19804b11438e182ed8d5f4b5a6ff503a"
server_key_iv: "1617cb7eef50aa183f1d1e763afb5598"
server_key_version: "encryptionkey@0"
I20260926 09:36:20.901312 17712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:20.916993 17712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:20.918825 17712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:20.918973 17712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:20.919175 17712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:20.919535 17712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:20.919589 17712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.919623 17712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:20.919651 17712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:20.925289 17712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34391
I20260926 09:36:20.925326 17840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34391 every 8 connection(s)
I20260926 09:36:20.925639 17712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:36:20.926874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 17712
I20260926 09:36:20.926980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:36:20.927140 392 external_mini_cluster.cc:1428] Setting key 33aa613b69a43204f2ffde9f8cd57a10
Sep 26 09:36:20 dist-test-slave-21h8 krb5kdc[17337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:20.934834 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.926359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48931 (local address 127.0.98.62:41775)
0926 09:36:20.926440 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:20.926442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:20.926591 (+ 149us) server_negotiation.cc:407] Connection header received
0926 09:36:20.927741 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:20.927743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:20.927754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:20.927821 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:20.928517 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.929054 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.929969 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.930303 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.930370 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.932535 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:20.932536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:20.932537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:20.932542 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:20.933764 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.934108 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.934109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.934109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.934158 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.934369 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.934369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.934369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.934461 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:20.934513 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:20.934718 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 09:36:20.934774 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 09:36:20.934978 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.926315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41775 (local address 127.0.98.3:48931)
0926 09:36:20.926479 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:20.926512 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:20.927660 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:20.927891 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:20.927895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:20.928250 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:20.928430 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.928434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.929128 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.929128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.929909 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.929911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.930399 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.930400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.930504 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.930962 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:20.930969 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:20.932452 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:20.933834 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:20.933840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:20.933841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:20.934044 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:20.934224 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:20.934225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:20.934225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:20.934316 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:20.934513 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:20.934516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:20.934617 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 09:36:20.934691 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:36:20.934800 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260926 09:36:20.935436 17841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41775
I20260926 09:36:20.935523 17841 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:20.935648 17841 heartbeater.cc:511] Master 127.0.98.62:41775 requested a full tablet report, sending...
I20260926 09:36:20.936012 17382 ts_manager.cc:194] Registered new tserver with Master: 0a451b7941d3449c876ffb1e2fcf379c (127.0.98.3:34391)
I20260926 09:36:20.936349 17382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48931
I20260926 09:36:20.938897 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:36:20.945659 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.940206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.0.98.62:41775)
0926 09:36:20.940272 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:20.940274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:20.940330 (+ 56us) server_negotiation.cc:407] Connection header received
0926 09:36:20.940364 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:20.940364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:20.940372 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:20.940417 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:20.940901 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.941369 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.942066 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.942246 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.942306 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.943469 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:20.943471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:20.943471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:20.943477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:20.944642 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.944947 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.944948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.944949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.945003 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:20.945168 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:20.945168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:20.945169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:20.945294 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:20.945364 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:20.945464 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 09:36:20.945531 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:36:20.947369 17382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:36:20.949057 17382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:36:20.955305 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.951403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34391 (local address 127.0.0.1:60518)
0926 09:36:20.951547 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:20.951584 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:20.951648 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:20.951954 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:20.951957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:20.951960 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:20.952037 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.952038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.953350 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.953350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.954233 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.954237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.955026 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.955027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.955126 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.955141 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:36:20.955185 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:36:20.955233 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":42,"threads_started":1}
I20260926 09:36:20.955523 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.951403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45043 (local address 127.0.0.1:34482)
0926 09:36:20.951778 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:20.951789 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:20.951857 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:20.952107 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:20.952108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:20.952110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:20.952160 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.952162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.953185 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.953186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.954338 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.954340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.955204 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.955205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.955266 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.955271 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:36:20.955391 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 09:36:20.955438 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260926 09:36:20.955828 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.951603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.0.98.3:34391)
0926 09:36:20.951772 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:20.951778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:20.951789 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:36:20.951827 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:20.951830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:20.951846 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:20.951933 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:20.952162 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.953228 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.954327 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.954930 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.955534 (+ 604us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.955660 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:20.955685 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 09:36:20.955733 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260926 09:36:20.955837 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.951513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.0.98.2:45043)
0926 09:36:20.951719 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:20.951724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:20.951848 (+ 124us) server_negotiation.cc:407] Connection header received
0926 09:36:20.951956 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:20.951958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:20.951973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:20.952109 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:20.952254 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.953049 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.954485 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.955088 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.955206 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.955265 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:20.955688 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 09:36:20.955736 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
I20260926 09:36:20.955907 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.951410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.0.1:54346)
0926 09:36:20.952038 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:20.952049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:20.952096 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:20.952386 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:20.952386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:20.952389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:20.952430 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.952432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.953605 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.953606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.954666 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:20.954668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.955666 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.955668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:20.955727 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.955731 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:20.955789 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:36:20.955827 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":94,"threads_started":1}
I20260926 09:36:20.955969 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:20.951666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.0.98.1:38433)
0926 09:36:20.951841 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:20.951845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:20.952153 (+ 308us) server_negotiation.cc:407] Connection header received
0926 09:36:20.952210 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:20.952212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:20.952228 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:20.952297 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:20.952503 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.953498 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.954753 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:20.955508 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:20.955569 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:20.955630 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:20.955823 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 09:36:20.955871 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260926 09:36:20.956501 17641 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3bf278e56147c68367862fbdcc5dc6 (DEFAULT_TABLE table=test-table [id=2379d2efa78b4431bacb36f139af1ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:20.956725 17507 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3bf278e56147c68367862fbdcc5dc6 (DEFAULT_TABLE table=test-table [id=2379d2efa78b4431bacb36f139af1ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:20.956858 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3bf278e56147c68367862fbdcc5dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:20.957023 17507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3bf278e56147c68367862fbdcc5dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:20.956976 17775 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3bf278e56147c68367862fbdcc5dc6 (DEFAULT_TABLE table=test-table [id=2379d2efa78b4431bacb36f139af1ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:20.957217 17775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3bf278e56147c68367862fbdcc5dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:20.959987 17858 tablet_bootstrap.cc:492] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Bootstrap starting.
I20260926 09:36:20.960305 17859 tablet_bootstrap.cc:492] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1: Bootstrap starting.
I20260926 09:36:20.960527 17860 tablet_bootstrap.cc:492] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Bootstrap starting.
I20260926 09:36:20.960698 17858 tablet_bootstrap.cc:654] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:20.961207 17860 tablet_bootstrap.cc:654] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:20.961308 17858 log.cc:824] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:20.961793 17860 log.cc:824] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:20.962124 17858 tablet_bootstrap.cc:492] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: No bootstrap required, opened a new log
I20260926 09:36:20.962193 17859 tablet_bootstrap.cc:654] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:20.962204 17858 ts_tablet_manager.cc:1397] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:20.962538 17860 tablet_bootstrap.cc:492] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: No bootstrap required, opened a new log
I20260926 09:36:20.962608 17860 ts_tablet_manager.cc:1397] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:20.962698 17859 log.cc:824] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:20.963546 17859 tablet_bootstrap.cc:492] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1: No bootstrap required, opened a new log
I20260926 09:36:20.963625 17859 ts_tablet_manager.cc:1397] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:36:20.964205 17858 raft_consensus.cc:348] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:20.964393 17858 raft_consensus.cc:374] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:20.964430 17858 raft_consensus.cc:729] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a451b7941d3449c876ffb1e2fcf379c, State: Initialized, Role: FOLLOWER
I20260926 09:36:20.964560 17858 consensus_queue.cc:260] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:20.964641 17860 raft_consensus.cc:348] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:20.964810 17860 raft_consensus.cc:374] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:20.964834 17858 ts_tablet_manager.cc:1428] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:20.964865 17860 raft_consensus.cc:729] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbf60093076461dac24a117ef33b6d4, State: Initialized, Role: FOLLOWER
I20260926 09:36:20.964939 17841 heartbeater.cc:503] Master 127.0.98.62:41775 was elected leader, sending a full tablet report...
I20260926 09:36:20.965018 17860 consensus_queue.cc:260] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:20.965309 17860 ts_tablet_manager.cc:1428] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:20.965626 17707 heartbeater.cc:503] Master 127.0.98.62:41775 was elected leader, sending a full tablet report...
I20260926 09:36:20.966089 17859 raft_consensus.cc:348] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:20.966277 17859 raft_consensus.cc:374] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:20.966334 17859 raft_consensus.cc:729] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174bc08df2724fb8a2fd1d5e2e1316a1, State: Initialized, Role: FOLLOWER
I20260926 09:36:20.966444 17859 consensus_queue.cc:260] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:20.966763 17859 ts_tablet_manager.cc:1428] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:36:20.966769 17573 heartbeater.cc:503] Master 127.0.98.62:41775 was elected leader, sending a full tablet report...
W20260926 09:36:21.024667 17708 tablet.cc:2369] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:36:21.102664 17574 tablet.cc:2369] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:36:21.176206 17842 tablet.cc:2369] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:36:21.177327 17864 raft_consensus.cc:482] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:21.177414 17864 raft_consensus.cc:504] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:21.177772 17864 leader_election.cc:288] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbf60093076461dac24a117ef33b6d4 (127.0.98.2:45043), 174bc08df2724fb8a2fd1d5e2e1316a1 (127.0.98.1:38433)
I20260926 09:36:21.180776 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.178042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34201 (local address 127.0.98.2:45043)
0926 09:36:21.178126 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.178129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.178140 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:36:21.178176 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.178177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.178189 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.178269 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:21.178430 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.178946 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.180023 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.180400 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.180490 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.180500 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.180657 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.180698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:36:21.180781 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.178053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56969 (local address 127.0.98.1:38433)
0926 09:36:21.178102 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.178104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.178236 (+ 132us) server_negotiation.cc:407] Connection header received
0926 09:36:21.178298 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.178300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.178310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.178380 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:21.178527 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.179050 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.180063 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.180417 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.180499 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.180508 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.180673 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.180716 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:36:21.180789 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.177965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.98.3:56969)
0926 09:36:21.178173 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.178180 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.178247 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.178382 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.178383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.178384 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:21.178457 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.178463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.179128 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.179129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.179973 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.179975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.180500 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.180501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.180583 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.180586 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.180655 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.180697 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":140,"threads_started":1}
I20260926 09:36:21.180789 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.177926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45043 (local address 127.0.98.3:34201)
0926 09:36:21.177981 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.177993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.178032 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.178273 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.178274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.178276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:21.178352 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.178354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.179047 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.179049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.179921 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.179923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.180489 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.180490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.180573 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.180576 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.180651 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.180700 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":20}
I20260926 09:36:21.181313 17527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3bf278e56147c68367862fbdcc5dc6" candidate_uuid: "0a451b7941d3449c876ffb1e2fcf379c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" is_pre_election: true
I20260926 09:36:21.181411 17661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3bf278e56147c68367862fbdcc5dc6" candidate_uuid: "0a451b7941d3449c876ffb1e2fcf379c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbf60093076461dac24a117ef33b6d4" is_pre_election: true
I20260926 09:36:21.181485 17527 raft_consensus.cc:2454] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a451b7941d3449c876ffb1e2fcf379c in term 0.
I20260926 09:36:21.181607 17661 raft_consensus.cc:2454] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a451b7941d3449c876ffb1e2fcf379c in term 0.
I20260926 09:36:21.181725 17730 leader_election.cc:302] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a451b7941d3449c876ffb1e2fcf379c, 174bc08df2724fb8a2fd1d5e2e1316a1; no voters:
I20260926 09:36:21.181916 17864 raft_consensus.cc:2781] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:36:21.181973 17864 raft_consensus.cc:482] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:36:21.181993 17864 raft_consensus.cc:3037] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:21.182739 17864 raft_consensus.cc:504] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:21.182897 17864 leader_election.cc:288] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [CANDIDATE]: Term 1 election: Requested vote from peers fbbf60093076461dac24a117ef33b6d4 (127.0.98.2:45043), 174bc08df2724fb8a2fd1d5e2e1316a1 (127.0.98.1:38433)
I20260926 09:36:21.183053 17661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3bf278e56147c68367862fbdcc5dc6" candidate_uuid: "0a451b7941d3449c876ffb1e2fcf379c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbf60093076461dac24a117ef33b6d4"
I20260926 09:36:21.183125 17661 raft_consensus.cc:3037] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:21.183931 17661 raft_consensus.cc:2454] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a451b7941d3449c876ffb1e2fcf379c in term 1.
I20260926 09:36:21.184124 17728 leader_election.cc:302] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a451b7941d3449c876ffb1e2fcf379c, fbbf60093076461dac24a117ef33b6d4; no voters:
I20260926 09:36:21.184089 17527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3bf278e56147c68367862fbdcc5dc6" candidate_uuid: "0a451b7941d3449c876ffb1e2fcf379c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1"
I20260926 09:36:21.184182 17527 raft_consensus.cc:3037] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:21.184239 17864 raft_consensus.cc:2781] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:21.184389 17864 raft_consensus.cc:686] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 1 LEADER]: Becoming Leader. State: Replica: 0a451b7941d3449c876ffb1e2fcf379c, State: Running, Role: LEADER
I20260926 09:36:21.184492 17864 consensus_queue.cc:237] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260926 09:36:21.185135 17527 raft_consensus.cc:2454] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a451b7941d3449c876ffb1e2fcf379c in term 1.
I20260926 09:36:21.185734 17381 catalog_manager.cc:5495] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a451b7941d3449c876ffb1e2fcf379c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0a451b7941d3449c876ffb1e2fcf379c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a451b7941d3449c876ffb1e2fcf379c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } health_report { overall_health: UNKNOWN } } }
I20260926 09:36:21.237470 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.235186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.0.98.3:34391)
0926 09:36:21.235256 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.235258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.235266 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:36:21.235306 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.235347 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.235369 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.235443 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:21.235580 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.236039 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.236808 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.236986 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.237034 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.237180 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:21.237280 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:21.237339 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.237380 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.237396 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:36:21.239482 17527 raft_consensus.cc:1261] T 9e3bf278e56147c68367862fbdcc5dc6 P 174bc08df2724fb8a2fd1d5e2e1316a1 [term 1 FOLLOWER]: Refusing update from remote peer 0a451b7941d3449c876ffb1e2fcf379c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:21.239511 17661 raft_consensus.cc:1261] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 1 FOLLOWER]: Refusing update from remote peer 0a451b7941d3449c876ffb1e2fcf379c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:21.239861 17864 consensus_queue.cc:1035] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [LEADER]: Connected to new peer: Peer: permanent_uuid: "174bc08df2724fb8a2fd1d5e2e1316a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:21.240056 17868 consensus_queue.cc:1035] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbf60093076461dac24a117ef33b6d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:21.242703 17872 mvcc.cc:205] Tried to move back new op lower bound from 7333541401553440768 to 7333541401332269056. Current Snapshot: MvccSnapshot[applied={T|T < 7333541401553440768}]
I20260926 09:36:21.243544 17871 mvcc.cc:205] Tried to move back new op lower bound from 7333541401553440768 to 7333541401332269056. Current Snapshot: MvccSnapshot[applied={T|T < 7333541401553440768 or (T in {7333541401553440768})}]
I20260926 09:36:21.244705 17382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:21.244807 17382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:21.245532 17382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c22dab149e6491d889473745a527287: Sending DeleteTablet for 3 replicas of tablet 9e3bf278e56147c68367862fbdcc5dc6
I20260926 09:36:21.245895 17775 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3bf278e56147c68367862fbdcc5dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60518
I20260926 09:36:21.246057 17507 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3bf278e56147c68367862fbdcc5dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54346
I20260926 09:36:21.246117 17641 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3bf278e56147c68367862fbdcc5dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34482
I20260926 09:36:21.246196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 17444
I20260926 09:36:21.246778 17882 tablet_replica.cc:330] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: stopping tablet replica
I20260926 09:36:21.246881 17882 raft_consensus.cc:2229] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:36:21.246982 17882 raft_consensus.cc:2258] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:21.250514 17880 tablet_replica.cc:330] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: stopping tablet replica
I20260926 09:36:21.250658 17880 raft_consensus.cc:2229] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:36:21.250820 17880 raft_consensus.cc:2258] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:21.251343 17880 ts_tablet_manager.cc:1905] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:21.251547 17882 ts_tablet_manager.cc:1905] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:21.253379 17882 ts_tablet_manager.cc:1918] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:36:21.253403 17880 ts_tablet_manager.cc:1918] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:36:21.253465 17880 log.cc:1196] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/9e3bf278e56147c68367862fbdcc5dc6
I20260926 09:36:21.253517 17882 log.cc:1196] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/9e3bf278e56147c68367862fbdcc5dc6
I20260926 09:36:21.253793 17882 ts_tablet_manager.cc:1939] T 9e3bf278e56147c68367862fbdcc5dc6 P fbbf60093076461dac24a117ef33b6d4: Deleting consensus metadata
I20260926 09:36:21.253893 17880 ts_tablet_manager.cc:1939] T 9e3bf278e56147c68367862fbdcc5dc6 P 0a451b7941d3449c876ffb1e2fcf379c: Deleting consensus metadata
I20260926 09:36:21.254258 17366 catalog_manager.cc:4841] TS fbbf60093076461dac24a117ef33b6d4 (127.0.98.2:45043): tablet 9e3bf278e56147c68367862fbdcc5dc6 (table test-table [id=2379d2efa78b4431bacb36f139af1ec8]) successfully deleted
W20260926 09:36:21.254309 17368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38433 (error 108)
I20260926 09:36:21.254371 17367 catalog_manager.cc:4841] TS 0a451b7941d3449c876ffb1e2fcf379c (127.0.98.3:34391): tablet 9e3bf278e56147c68367862fbdcc5dc6 (table test-table [id=2379d2efa78b4431bacb36f139af1ec8]) successfully deleted
I20260926 09:36:21.254805 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.254691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.0.1:54360)
0926 09:36:21.254737 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.254769 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 09:36:21.254890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 17578
W20260926 09:36:21.254913 17368 catalog_manager.cc:4584] TS 174bc08df2724fb8a2fd1d5e2e1316a1 (127.0.98.1:38433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e3bf278e56147c68367862fbdcc5dc6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38433: connect: Connection refused (error 111)
I20260926 09:36:21.259747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 17712
I20260926 09:36:21.263782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 17350
2026-09-26T09:36:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2155 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T09:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:21Z Disabled control of system clock
I20260926 09:36:21.290798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41695
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:36:21.373148 17887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:21.373351 17887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:21.373371 17887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:21.374600 17887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:21.374639 17887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:21.374651 17887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:21.374665 17887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:21.374676 17887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:21.375936 17887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:21.376108 17887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:21.376315 17887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:21.381196 17893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:21.381207 17892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:21.381227 17895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:21.381472 17887 server_base.cc:1034] running on GCE node
I20260926 09:36:21.381693 17887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:21.381908 17887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:21.383065 17887 hybrid_clock.cc:648] HybridClock initialized: now 1790415381383047 us; error 33 us; skew 500 ppm
I20260926 09:36:21.384411 17887 webserver.cc:460] Webserver started at http://127.0.98.62:34383/ using document root <none> and password file <none>
I20260926 09:36:21.384647 17887 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:21.384693 17887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:21.384842 17887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:21.385892 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "4c2aceea5bc249488367896450b378b3"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "4e795895c9d8cdd74167b118696e1f31"
server_key_iv: "a58b35dbea94e107ab0843a6db91036f"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.386266 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c2aceea5bc249488367896450b378b3"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "4e795895c9d8cdd74167b118696e1f31"
server_key_iv: "a58b35dbea94e107ab0843a6db91036f"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.387633 17887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:21.388386 17901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.388547 17887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:21.388672 17887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "4c2aceea5bc249488367896450b378b3"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "4e795895c9d8cdd74167b118696e1f31"
server_key_iv: "a58b35dbea94e107ab0843a6db91036f"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.388756 17887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:21.412144 17887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:21.412439 17887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:21.412588 17887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:21.416553 17887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41695
I20260926 09:36:21.416582 17953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41695 every 8 connection(s)
I20260926 09:36:21.416901 17887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:21.417783 17954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:21.420017 17954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3: Bootstrap starting.
I20260926 09:36:21.420565 17954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:21.420850 17954 log.cc:824] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:21.421484 17954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3: No bootstrap required, opened a new log
I20260926 09:36:21.423008 17954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aceea5bc249488367896450b378b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41695 } }
I20260926 09:36:21.423133 17954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:21.423154 17954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2aceea5bc249488367896450b378b3, State: Initialized, Role: FOLLOWER
I20260926 09:36:21.423374 17954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aceea5bc249488367896450b378b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41695 } }
I20260926 09:36:21.423439 17954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:21.423460 17954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:21.423523 17954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:21.423978 17954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aceea5bc249488367896450b378b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41695 } }
I20260926 09:36:21.424089 17954 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2aceea5bc249488367896450b378b3; no voters:
I20260926 09:36:21.424260 17954 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:21.424311 17957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:21.424494 17954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:21.424540 17957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2aceea5bc249488367896450b378b3, State: Running, Role: LEADER
I20260926 09:36:21.424638 17957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aceea5bc249488367896450b378b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41695 } }
I20260926 09:36:21.424975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 17887
I20260926 09:36:21.425076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:21.425122 17959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2aceea5bc249488367896450b378b3. Latest consensus state: current_term: 1 leader_uuid: "4c2aceea5bc249488367896450b378b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aceea5bc249488367896450b378b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41695 } } }
I20260926 09:36:21.425192 17959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:21.425292 392 external_mini_cluster.cc:1428] Setting key 645372bfe3f2e7fd6b4d9b324344351b
I20260926 09:36:21.425347 17958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2aceea5bc249488367896450b378b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aceea5bc249488367896450b378b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41695 } } }
I20260926 09:36:21.425625 17958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:21.425951 17966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:21.426625 17966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:21.428310 17966 catalog_manager.cc:1347] Generated new cluster ID: c31228aaf54d49faafbf9945bb8e662f
I20260926 09:36:21.428361 17966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:21.431188 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.425710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.0.98.62:41695)
0926 09:36:21.426057 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.426067 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.426097 (+ 30us) server_negotiation.cc:407] Connection header received
0926 09:36:21.426213 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.426219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.426287 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.426431 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:21.426716 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.427966 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.429063 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.429635 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.429747 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.430434 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:21.430437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:21.430438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:21.430625 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:21.430662 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:21.430674 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:21.430753 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.430834 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.430993 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":65,"threads_started":1}
I20260926 09:36:21.440513 17966 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:21.441162 17966 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:21.447480 17966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3: Generated new TSK 0
I20260926 09:36:21.447670 17966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:21.452926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41695
--builtin_ntp_servers=127.0.98.20:41175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:36:21.532469 17978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:21.532660 17978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:21.532681 17978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:21.533825 17978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:21.533864 17978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:21.533890 17978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:36:21.535197 17978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:21.535449 17978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:21.535739 17978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:21.541626 17984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:21.541683 17978 server_base.cc:1034] running on GCE node
W20260926 09:36:21.541778 17986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:21.541615 17983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:21.542089 17978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:21.542347 17978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:21.543502 17978 hybrid_clock.cc:648] HybridClock initialized: now 1790415381543491 us; error 27 us; skew 500 ppm
I20260926 09:36:21.544808 17978 webserver.cc:460] Webserver started at http://127.0.98.1:39863/ using document root <none> and password file <none>
I20260926 09:36:21.545058 17978 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:21.545111 17978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:21.545230 17978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:21.546509 17978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc75c678f15845f3bc0fb33f8bec8f19"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "37392e8beb35f2f9d222574da80ed4a0"
server_key_iv: "4f18c406ed697cbbfa4823751053bab2"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.546871 17978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc75c678f15845f3bc0fb33f8bec8f19"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "37392e8beb35f2f9d222574da80ed4a0"
server_key_iv: "4f18c406ed697cbbfa4823751053bab2"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.548507 17978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:21.549296 17992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.549464 17978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:21.549563 17978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "cc75c678f15845f3bc0fb33f8bec8f19"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "37392e8beb35f2f9d222574da80ed4a0"
server_key_iv: "4f18c406ed697cbbfa4823751053bab2"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.549642 17978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:21.565615 17978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:21.565883 17978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:21.566026 17978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:21.566236 17978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:21.566535 17978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:21.566567 17978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.566587 17978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:21.566633 17978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.572320 17978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33983
I20260926 09:36:21.572427 18105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33983 every 8 connection(s)
I20260926 09:36:21.572701 17978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:36:21.577663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 17978
I20260926 09:36:21.577767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:36:21.577762 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.573458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56687 (local address 127.0.98.62:41695)
0926 09:36:21.573517 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.573519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.573749 (+ 230us) server_negotiation.cc:407] Connection header received
0926 09:36:21.574303 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.574305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.574317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.574377 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:21.574996 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.575562 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.576500 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.576700 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.576754 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.577466 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:21.577467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:21.577467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:21.577472 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:21.577487 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:21.577491 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:21.577545 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.577659 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.577681 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:36:21.577975 392 external_mini_cluster.cc:1428] Setting key 1d1304a1c11fd8d3f8087d678224fe8a
I20260926 09:36:21.577983 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.573439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41695 (local address 127.0.98.1:56687)
0926 09:36:21.573619 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.573650 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.574161 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.574429 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.574433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.574440 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:21.574915 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.574919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.575644 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.575645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.576419 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.576421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.576773 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.576774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.576874 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.577331 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:21.577332 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:21.577357 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:21.577378 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:21.577566 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:21.577573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:21.577575 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.577656 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.577766 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260926 09:36:21.578492 18106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41695
I20260926 09:36:21.578594 18106 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:21.578768 18106 heartbeater.cc:511] Master 127.0.98.62:41695 requested a full tablet report, sending...
I20260926 09:36:21.579224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41695
--builtin_ntp_servers=127.0.98.20:41175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:36:21.579380 17917 ts_manager.cc:194] Registered new tserver with Master: cc75c678f15845f3bc0fb33f8bec8f19 (127.0.98.1:33983)
I20260926 09:36:21.580233 17917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56687
W20260926 09:36:21.660737 18109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:21.660930 18109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:21.660961 18109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:21.662072 18109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:21.662109 18109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:21.662137 18109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:36:21.663426 18109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:21.663610 18109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:21.663834 18109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:21.668990 18117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:21.668926 18114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:21.669184 18109 server_base.cc:1034] running on GCE node
W20260926 09:36:21.668948 18115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:21.669428 18109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:21.669663 18109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:21.670828 18109 hybrid_clock.cc:648] HybridClock initialized: now 1790415381670820 us; error 49 us; skew 500 ppm
I20260926 09:36:21.672014 18109 webserver.cc:460] Webserver started at http://127.0.98.2:42703/ using document root <none> and password file <none>
I20260926 09:36:21.672267 18109 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:21.672336 18109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:21.672487 18109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:21.673487 18109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "19a991dd8db4480588ce8aae862204ec"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "9e91f1d6fa7a2c6c166b65a6a5bbf4e1"
server_key_iv: "95a5848b5b9e324aa0546ff464db71b7"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.673846 18109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "19a991dd8db4480588ce8aae862204ec"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "9e91f1d6fa7a2c6c166b65a6a5bbf4e1"
server_key_iv: "95a5848b5b9e324aa0546ff464db71b7"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.675227 18109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:21.676026 18123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.676203 18109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:21.676301 18109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "19a991dd8db4480588ce8aae862204ec"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "9e91f1d6fa7a2c6c166b65a6a5bbf4e1"
server_key_iv: "95a5848b5b9e324aa0546ff464db71b7"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.676373 18109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:21.717428 18109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:21.717772 18109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:21.717959 18109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:21.718202 18109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:21.718545 18109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:21.718608 18109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.718649 18109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:21.718677 18109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.724284 18109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40241
I20260926 09:36:21.724336 18236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40241 every 8 connection(s)
I20260926 09:36:21.724735 18109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:36:21.725028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 18109
I20260926 09:36:21.725143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:36:21.725360 392 external_mini_cluster.cc:1428] Setting key b4bbdbfcd05006463c414f8c8f91decb
I20260926 09:36:21.726416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41695
--builtin_ntp_servers=127.0.98.20:41175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:36:21.730746 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.725667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35333 (local address 127.0.98.62:41695)
0926 09:36:21.725723 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.725725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.725970 (+ 245us) server_negotiation.cc:407] Connection header received
0926 09:36:21.726627 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.726628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.726640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.726669 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:21.727471 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.728326 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.729270 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.729482 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.729541 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.730448 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:21.730449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:21.730450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:21.730453 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:21.730466 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:21.730470 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:21.730549 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.730656 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.730679 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:36:21.731048 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.725658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41695 (local address 127.0.98.2:35333)
0926 09:36:21.725859 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.725894 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.726487 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.726821 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.726824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.726832 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:21.727306 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.727372 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.728425 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.728426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.729177 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.729178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.729575 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.729576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.729744 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.730293 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:21.730294 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:21.730332 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:21.730359 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:21.730563 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:21.730570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:21.730573 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.730650 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.730789 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260926 09:36:21.731611 18237 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41695
I20260926 09:36:21.731738 18237 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:21.731945 18237 heartbeater.cc:511] Master 127.0.98.62:41695 requested a full tablet report, sending...
I20260926 09:36:21.732302 17917 ts_manager.cc:194] Registered new tserver with Master: 19a991dd8db4480588ce8aae862204ec (127.0.98.2:40241)
I20260926 09:36:21.732666 17917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35333
W20260926 09:36:21.807726 18240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:21.807909 18240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:21.807929 18240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:21.809052 18240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:21.809093 18240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:21.809120 18240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:36:21.810487 18240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:21.810724 18240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:21.810986 18240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:21.815677 18245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:21.815689 18248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:21.815688 18246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:21.815846 18240 server_base.cc:1034] running on GCE node
I20260926 09:36:21.816067 18240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:21.816246 18240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:21.817389 18240 hybrid_clock.cc:648] HybridClock initialized: now 1790415381817374 us; error 30 us; skew 500 ppm
I20260926 09:36:21.818655 18240 webserver.cc:460] Webserver started at http://127.0.98.3:40273/ using document root <none> and password file <none>
I20260926 09:36:21.818892 18240 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:21.818935 18240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:21.819080 18240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:21.820142 18240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "6499d8d3cbc943bfb28a1bab65e56b51"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "164ded9a1ff81e0faab7c5a49515459d"
server_key_iv: "953223382132f5cc59097b222d9d8d28"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.820469 18240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6499d8d3cbc943bfb28a1bab65e56b51"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "164ded9a1ff81e0faab7c5a49515459d"
server_key_iv: "953223382132f5cc59097b222d9d8d28"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.821681 18240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:36:21.822341 18254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.822527 18240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:21.822618 18240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "6499d8d3cbc943bfb28a1bab65e56b51"
format_stamp: "Formatted at 2026-09-26 09:36:21 on dist-test-slave-21h8"
server_key: "164ded9a1ff81e0faab7c5a49515459d"
server_key_iv: "953223382132f5cc59097b222d9d8d28"
server_key_version: "encryptionkey@0"
I20260926 09:36:21.822695 18240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:21.852389 18240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:21.852739 18240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:21.852887 18240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:21.853116 18240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:21.853446 18240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:21.853505 18240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.853543 18240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:21.853569 18240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:21.858843 18240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39749
I20260926 09:36:21.858883 18367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39749 every 8 connection(s)
I20260926 09:36:21.859186 18240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:36:21.862149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 18240
I20260926 09:36:21.862258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:36:21.862438 392 external_mini_cluster.cc:1428] Setting key 3c67c7b035d23425809def8ebf3f6fb7
I20260926 09:36:21.864502 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.859966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40025 (local address 127.0.98.62:41695)
0926 09:36:21.860034 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.860035 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.860258 (+ 223us) server_negotiation.cc:407] Connection header received
0926 09:36:21.860714 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.860715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.860725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.860791 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:21.861401 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.861884 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.863056 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.863369 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.863434 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.864292 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:21.864293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:21.864293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:21.864297 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:21.864308 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:21.864311 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:21.864346 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.864425 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.864447 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:36:21.864677 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.859950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41695 (local address 127.0.98.3:40025)
0926 09:36:21.860131 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.860171 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.860635 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.860845 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.860848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.860857 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:21.861334 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.861339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.861962 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.861963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.862975 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.862978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.863437 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.863438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.863562 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.864164 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:21.864165 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:21.864198 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:21.864220 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:21.864354 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:21.864357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:21.864359 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.864416 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.864484 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260926 09:36:21.865120 18368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41695
I20260926 09:36:21.865222 18368 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:21.865410 18368 heartbeater.cc:511] Master 127.0.98.62:41695 requested a full tablet report, sending...
I20260926 09:36:21.865756 17917 ts_manager.cc:194] Registered new tserver with Master: 6499d8d3cbc943bfb28a1bab65e56b51 (127.0.98.3:39749)
I20260926 09:36:21.866055 17917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40025
I20260926 09:36:21.874137 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:36:21.878484 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.875605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.0.98.62:41695)
0926 09:36:21.875673 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.875675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.875775 (+ 100us) server_negotiation.cc:407] Connection header received
0926 09:36:21.875849 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.875850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.875863 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.875932 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:21.876123 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.876576 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.877301 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.877488 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.877558 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.878221 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:21.878223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:21.878223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:21.878227 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:21.878240 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:21.878244 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:21.878300 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.878352 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.878376 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:36:21.880054 17917 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:36:21.881515 17917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:36:21.887753 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.883641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40241 (local address 127.0.0.1:55114)
0926 09:36:21.884041 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.884051 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.884138 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.884316 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.884317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.884318 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:21.884346 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.884347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.885353 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.885354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.886622 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.886625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.887456 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.887457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.887553 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.887567 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.887637 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.887682 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":54,"threads_started":1}
I20260926 09:36:21.887764 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.883799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.0.98.2:40241)
0926 09:36:21.883991 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.883995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.884134 (+ 139us) server_negotiation.cc:407] Connection header received
0926 09:36:21.884220 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.884223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.884240 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.884322 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:21.884392 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.885250 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.886718 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.887380 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.887445 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.887545 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.887651 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.887692 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260926 09:36:21.888091 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.883638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39749 (local address 127.0.0.1:41192)
0926 09:36:21.884231 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.884241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.884404 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.884631 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.884632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.884635 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:21.884674 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.884675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.885890 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.885891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.886907 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.886909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.887860 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.887861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.887933 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.887937 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.887982 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.888016 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":23,"thread_start_us":77,"threads_started":1}
I20260926 09:36:21.888342 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.883843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.0.98.3:39749)
0926 09:36:21.883986 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.883991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.884376 (+ 385us) server_negotiation.cc:407] Connection header received
0926 09:36:21.884486 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.884488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.884503 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.884580 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:21.884767 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.885809 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.886976 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.887682 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.887741 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.887820 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.888214 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.888255 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260926 09:36:21.888578 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.883638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33983 (local address 127.0.0.1:36498)
0926 09:36:21.883789 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.883825 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.883888 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.884144 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.884146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.884148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:21.884205 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.884206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.885751 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.885752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.886836 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.886837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.888082 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.888083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.888278 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.888282 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.888329 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.888374 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":34,"thread_start_us":94,"threads_started":1}
I20260926 09:36:21.888482 18171 tablet_service.cc:1467] Processing CreateTablet for tablet 7f75bb982c08479ea92a3e34474b7284 (DEFAULT_TABLE table=test-table [id=bb10747f335c4ce6a967f3a5209b1121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:21.888831 18171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f75bb982c08479ea92a3e34474b7284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:21.888962 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.883791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.0.98.1:33983)
0926 09:36:21.883984 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.883988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.883998 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:21.884040 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.884042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.884053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.884139 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:21.884261 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.885652 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.886915 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.887657 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.887742 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.888787 (+ 1045us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.888812 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.888855 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260926 09:36:21.888934 18302 tablet_service.cc:1467] Processing CreateTablet for tablet 7f75bb982c08479ea92a3e34474b7284 (DEFAULT_TABLE table=test-table [id=bb10747f335c4ce6a967f3a5209b1121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:21.889251 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f75bb982c08479ea92a3e34474b7284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:21.891637 18040 tablet_service.cc:1467] Processing CreateTablet for tablet 7f75bb982c08479ea92a3e34474b7284 (DEFAULT_TABLE table=test-table [id=bb10747f335c4ce6a967f3a5209b1121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:21.891993 18040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f75bb982c08479ea92a3e34474b7284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:21.892274 18382 tablet_bootstrap.cc:492] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: Bootstrap starting.
I20260926 09:36:21.892688 18383 tablet_bootstrap.cc:492] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: Bootstrap starting.
I20260926 09:36:21.892938 18382 tablet_bootstrap.cc:654] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:21.893292 18383 tablet_bootstrap.cc:654] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:21.893280 18382 log.cc:824] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:21.893632 18383 log.cc:824] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:21.894158 18382 tablet_bootstrap.cc:492] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: No bootstrap required, opened a new log
I20260926 09:36:21.894245 18382 ts_tablet_manager.cc:1397] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:36:21.894304 18383 tablet_bootstrap.cc:492] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: No bootstrap required, opened a new log
I20260926 09:36:21.894368 18383 ts_tablet_manager.cc:1397] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:36:21.895430 18386 tablet_bootstrap.cc:492] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19: Bootstrap starting.
I20260926 09:36:21.896093 18386 tablet_bootstrap.cc:654] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:21.896392 18386 log.cc:824] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:21.896675 18382 raft_consensus.cc:348] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.896855 18382 raft_consensus.cc:374] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:21.896901 18382 raft_consensus.cc:729] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a991dd8db4480588ce8aae862204ec, State: Initialized, Role: FOLLOWER
I20260926 09:36:21.896785 18383 raft_consensus.cc:348] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.896941 18383 raft_consensus.cc:374] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:21.896991 18383 raft_consensus.cc:729] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6499d8d3cbc943bfb28a1bab65e56b51, State: Initialized, Role: FOLLOWER
I20260926 09:36:21.897029 18382 consensus_queue.cc:260] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.897101 18383 consensus_queue.cc:260] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.897207 18386 tablet_bootstrap.cc:492] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19: No bootstrap required, opened a new log
I20260926 09:36:21.897274 18386 ts_tablet_manager.cc:1397] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:21.897286 18382 ts_tablet_manager.cc:1428] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:21.897352 18383 ts_tablet_manager.cc:1428] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:21.897398 18237 heartbeater.cc:503] Master 127.0.98.62:41695 was elected leader, sending a full tablet report...
I20260926 09:36:21.897470 18368 heartbeater.cc:503] Master 127.0.98.62:41695 was elected leader, sending a full tablet report...
I20260926 09:36:21.899353 18386 raft_consensus.cc:348] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.899536 18386 raft_consensus.cc:374] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:21.899583 18386 raft_consensus.cc:729] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc75c678f15845f3bc0fb33f8bec8f19, State: Initialized, Role: FOLLOWER
I20260926 09:36:21.899694 18386 consensus_queue.cc:260] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.899966 18386 ts_tablet_manager.cc:1428] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:21.900022 18106 heartbeater.cc:503] Master 127.0.98.62:41695 was elected leader, sending a full tablet report...
I20260926 09:36:21.925184 18390 raft_consensus.cc:482] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:21.925359 18390 raft_consensus.cc:504] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.925792 18390 leader_election.cc:288] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19a991dd8db4480588ce8aae862204ec (127.0.98.2:40241), 6499d8d3cbc943bfb28a1bab65e56b51 (127.0.98.3:39749)
I20260926 09:36:21.929739 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.926124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39559 (local address 127.0.98.3:39749)
0926 09:36:21.926172 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.926174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.926181 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:36:21.926225 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.926226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.926233 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.926299 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:21.926441 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.927234 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.928729 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.929305 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.929367 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.929376 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.929623 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.929672 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:36:21.929739 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.926022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58779 (local address 127.0.98.2:40241)
0926 09:36:21.926063 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.926065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.926277 (+ 212us) server_negotiation.cc:407] Connection header received
0926 09:36:21.926378 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.926379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.926392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.926441 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:21.926580 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.927252 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.928737 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.929284 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.929366 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.929376 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.929623 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.929672 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:21.929740 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.925930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39749 (local address 127.0.98.1:39559)
0926 09:36:21.926031 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.926043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.926086 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.926304 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.926304 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.926306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:21.926373 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.926375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.927347 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.927348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.928624 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.928626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.929419 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.929420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.929523 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.929525 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.929586 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.929642 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":46}
I20260926 09:36:21.929747 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.925941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40241 (local address 127.0.98.1:58779)
0926 09:36:21.926193 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.926204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:21.926289 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:21.926455 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:21.926456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:21.926457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:21.926514 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.926515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.927338 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.927338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.928626 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:21.928627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.929421 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.929421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:21.929508 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.929513 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:36:21.929586 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:36:21.929644 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":63,"thread_start_us":79,"threads_started":1}
I20260926 09:36:21.930127 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f75bb982c08479ea92a3e34474b7284" candidate_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a991dd8db4480588ce8aae862204ec" is_pre_election: true
I20260926 09:36:21.930123 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f75bb982c08479ea92a3e34474b7284" candidate_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" is_pre_election: true
I20260926 09:36:21.930280 18322 raft_consensus.cc:2454] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc75c678f15845f3bc0fb33f8bec8f19 in term 0.
I20260926 09:36:21.930280 18191 raft_consensus.cc:2454] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc75c678f15845f3bc0fb33f8bec8f19 in term 0.
I20260926 09:36:21.930519 17996 leader_election.cc:302] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a991dd8db4480588ce8aae862204ec, cc75c678f15845f3bc0fb33f8bec8f19; no voters:
I20260926 09:36:21.930665 18390 raft_consensus.cc:2781] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:36:21.930748 18390 raft_consensus.cc:482] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:36:21.930768 18390 raft_consensus.cc:3037] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:21.931640 18390 raft_consensus.cc:504] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.931785 18390 leader_election.cc:288] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [CANDIDATE]: Term 1 election: Requested vote from peers 19a991dd8db4480588ce8aae862204ec (127.0.98.2:40241), 6499d8d3cbc943bfb28a1bab65e56b51 (127.0.98.3:39749)
I20260926 09:36:21.931941 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f75bb982c08479ea92a3e34474b7284" candidate_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a991dd8db4480588ce8aae862204ec"
I20260926 09:36:21.932013 18191 raft_consensus.cc:3037] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:21.932719 18191 raft_consensus.cc:2454] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc75c678f15845f3bc0fb33f8bec8f19 in term 1.
I20260926 09:36:21.932864 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f75bb982c08479ea92a3e34474b7284" candidate_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6499d8d3cbc943bfb28a1bab65e56b51"
I20260926 09:36:21.932920 17996 leader_election.cc:302] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a991dd8db4480588ce8aae862204ec, cc75c678f15845f3bc0fb33f8bec8f19; no voters:
I20260926 09:36:21.932953 18322 raft_consensus.cc:3037] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:21.933037 18390 raft_consensus.cc:2781] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:21.933203 18390 raft_consensus.cc:686] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 1 LEADER]: Becoming Leader. State: Replica: cc75c678f15845f3bc0fb33f8bec8f19, State: Running, Role: LEADER
I20260926 09:36:21.933298 18390 consensus_queue.cc:237] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } }
I20260926 09:36:21.933903 18322 raft_consensus.cc:2454] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc75c678f15845f3bc0fb33f8bec8f19 in term 1.
I20260926 09:36:21.934146 17917 catalog_manager.cc:5495] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc75c678f15845f3bc0fb33f8bec8f19 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc75c678f15845f3bc0fb33f8bec8f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 } health_report { overall_health: UNKNOWN } } }
I20260926 09:36:21.939194 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.936963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.0.98.1:33983)
0926 09:36:21.937037 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:21.937039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:21.937048 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:36:21.937073 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:21.937074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:21.937088 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:21.937147 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:21.937304 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.937750 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.938437 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:21.938615 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:21.938662 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:21.938790 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:21.938875 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:21.938932 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:21.938987 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 09:36:21.939002 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:36:21.940954 18191 raft_consensus.cc:1261] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 1 FOLLOWER]: Refusing update from remote peer cc75c678f15845f3bc0fb33f8bec8f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:21.940963 18322 raft_consensus.cc:1261] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 1 FOLLOWER]: Refusing update from remote peer cc75c678f15845f3bc0fb33f8bec8f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:21.941273 18390 consensus_queue.cc:1035] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a991dd8db4480588ce8aae862204ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:21.941433 18392 consensus_queue.cc:1035] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6499d8d3cbc943bfb28a1bab65e56b51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:21.943655 18395 mvcc.cc:205] Tried to move back new op lower bound from 7333541404427776000 to 7333541404399386624. Current Snapshot: MvccSnapshot[applied={T|T < 7333541404427776000}]
I20260926 09:36:21.943897 18396 mvcc.cc:205] Tried to move back new op lower bound from 7333541404427776000 to 7333541404399386624. Current Snapshot: MvccSnapshot[applied={T|T < 7333541404427776000}]
I20260926 09:36:21.944784 18397 mvcc.cc:205] Tried to move back new op lower bound from 7333541404427776000 to 7333541404399386624. Current Snapshot: MvccSnapshot[applied={T|T < 7333541404427776000}]
I20260926 09:36:21.946239 17917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:21.946334 17917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:21.947041 17917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c2aceea5bc249488367896450b378b3: Sending DeleteTablet for 3 replicas of tablet 7f75bb982c08479ea92a3e34474b7284
I20260926 09:36:21.947365 18040 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f75bb982c08479ea92a3e34474b7284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:21 UTC) from {username='slave'} at 127.0.0.1:36498
I20260926 09:36:21.947415 18171 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f75bb982c08479ea92a3e34474b7284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:21 UTC) from {username='slave'} at 127.0.0.1:55114
I20260926 09:36:21.947418 18302 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f75bb982c08479ea92a3e34474b7284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:21 UTC) from {username='slave'} at 127.0.0.1:41192
I20260926 09:36:21.947628 18407 tablet_replica.cc:330] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: stopping tablet replica
I20260926 09:36:21.947710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 17978
I20260926 09:36:21.947732 18407 raft_consensus.cc:2229] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:36:21.947731 18405 tablet_replica.cc:330] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19: stopping tablet replica
I20260926 09:36:21.947832 18407 raft_consensus.cc:2258] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:21.947948 18405 raft_consensus.cc:2229] T 7f75bb982c08479ea92a3e34474b7284 P cc75c678f15845f3bc0fb33f8bec8f19 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:36:21.948361 18406 tablet_replica.cc:330] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: stopping tablet replica
I20260926 09:36:21.948514 18406 raft_consensus.cc:2229] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:36:21.948645 18407 ts_tablet_manager.cc:1905] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:21.948655 18406 raft_consensus.cc:2258] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:21.949097 18406 ts_tablet_manager.cc:1905] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:21.950307 18407 ts_tablet_manager.cc:1918] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:36:21.950383 18407 log.cc:1196] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/7f75bb982c08479ea92a3e34474b7284
I20260926 09:36:21.950642 18407 ts_tablet_manager.cc:1939] T 7f75bb982c08479ea92a3e34474b7284 P 6499d8d3cbc943bfb28a1bab65e56b51: Deleting consensus metadata
I20260926 09:36:21.950878 18406 ts_tablet_manager.cc:1918] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:36:21.950977 18406 log.cc:1196] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/7f75bb982c08479ea92a3e34474b7284
I20260926 09:36:21.951041 17904 catalog_manager.cc:4841] TS 6499d8d3cbc943bfb28a1bab65e56b51 (127.0.98.3:39749): tablet 7f75bb982c08479ea92a3e34474b7284 (table test-table [id=bb10747f335c4ce6a967f3a5209b1121]) successfully deleted
I20260926 09:36:21.951256 18406 ts_tablet_manager.cc:1939] T 7f75bb982c08479ea92a3e34474b7284 P 19a991dd8db4480588ce8aae862204ec: Deleting consensus metadata
I20260926 09:36:21.951665 17905 catalog_manager.cc:4841] TS 19a991dd8db4480588ce8aae862204ec (127.0.98.2:40241): tablet 7f75bb982c08479ea92a3e34474b7284 (table test-table [id=bb10747f335c4ce6a967f3a5209b1121]) successfully deleted
W20260926 09:36:21.953832 17903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33983 (error 108)
I20260926 09:36:21.954290 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:21.954147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33983 (local address 127.0.0.1:36516)
0926 09:36:21.954213 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:21.954264 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:36:21.954324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 18109
W20260926 09:36:21.954432 17903 catalog_manager.cc:4584] TS cc75c678f15845f3bc0fb33f8bec8f19 (127.0.98.1:33983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f75bb982c08479ea92a3e34474b7284: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33983: connect: Connection refused (error 111)
I20260926 09:36:21.958516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 18240
I20260926 09:36:21.962882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 17887
2026-09-26T09:36:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (698 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:22 dist-test-slave-21h8 krb5kdc[18411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:22 dist-test-slave-21h8 krb5kdc[18411](info): set up 2 sockets
Sep 26 09:36:22 dist-test-slave-21h8 krb5kdc[18411](info): commencing operation
krb5kdc: starting...
W20260926 09:36:22.995010 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:23.120373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:23.200217 18424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:23.200444 18424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:23.200480 18424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:23.201622 18424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:23.201694 18424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:23.201719 18424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:23.201749 18424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:23.201781 18424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:23.203215 18424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:23.203504 18424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:23.203733 18424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:23.208527 18432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:23.208487 18430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:23.208504 18429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:23.208732 18424 server_base.cc:1034] running on GCE node
I20260926 09:36:23.208969 18424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:23.209259 18424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:23.210429 18424 hybrid_clock.cc:648] HybridClock initialized: now 1790415383210408 us; error 37 us; skew 500 ppm
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:23.213047 18424 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:23.213506 18424 webserver.cc:460] Webserver started at http://127.0.98.62:33711/ using document root <none> and password file <none>
I20260926 09:36:23.213753 18424 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:23.213800 18424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:23.213984 18424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:23.215018 18424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "8ec624a93a1c44a1a90abd82fe810346"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "8e8e126dcd63fe17ac4ac12dd6cd12d0"
server_key_iv: "9a29938bffad0a4242a734dbfe8599e7"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.215430 18424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ec624a93a1c44a1a90abd82fe810346"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "8e8e126dcd63fe17ac4ac12dd6cd12d0"
server_key_iv: "9a29938bffad0a4242a734dbfe8599e7"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.216830 18424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:36:23.217582 18439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.217777 18424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:23.217865 18424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "8ec624a93a1c44a1a90abd82fe810346"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "8e8e126dcd63fe17ac4ac12dd6cd12d0"
server_key_iv: "9a29938bffad0a4242a734dbfe8599e7"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.217937 18424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:23.259169 18424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:23.261153 18424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:23.261317 18424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:23.265457 18424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39091
I20260926 09:36:23.265518 18491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39091 every 8 connection(s)
I20260926 09:36:23.265884 18424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:23.266703 18492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:23.269057 18492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346: Bootstrap starting.
I20260926 09:36:23.269783 18492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:23.270211 18492 log.cc:824] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:23.270910 18492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346: No bootstrap required, opened a new log
I20260926 09:36:23.272589 18492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec624a93a1c44a1a90abd82fe810346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260926 09:36:23.272727 18492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:23.272750 18492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec624a93a1c44a1a90abd82fe810346, State: Initialized, Role: FOLLOWER
I20260926 09:36:23.272921 18492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec624a93a1c44a1a90abd82fe810346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260926 09:36:23.272991 18492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:23.273034 18492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:23.273083 18492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:23.273648 18492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec624a93a1c44a1a90abd82fe810346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260926 09:36:23.273759 18492 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec624a93a1c44a1a90abd82fe810346; no voters:
I20260926 09:36:23.273939 18492 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:23.273967 18495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:23.274246 18495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [term 1 LEADER]: Becoming Leader. State: Replica: 8ec624a93a1c44a1a90abd82fe810346, State: Running, Role: LEADER
I20260926 09:36:23.274338 18495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec624a93a1c44a1a90abd82fe810346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260926 09:36:23.274394 18492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:23.274761 18496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec624a93a1c44a1a90abd82fe810346. Latest consensus state: current_term: 1 leader_uuid: "8ec624a93a1c44a1a90abd82fe810346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec624a93a1c44a1a90abd82fe810346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } } }
I20260926 09:36:23.274847 18496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:23.274902 18497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec624a93a1c44a1a90abd82fe810346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec624a93a1c44a1a90abd82fe810346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } } }
I20260926 09:36:23.275012 18497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:23.275041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 18424
I20260926 09:36:23.275051 18499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:23.275162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:23.275382 392 external_mini_cluster.cc:1428] Setting key a4a43847e749d43d8660eb07fce738fa
I20260926 09:36:23.275584 18499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:23.278038 18499 catalog_manager.cc:1347] Generated new cluster ID: 4133ffd451e34721955f1c1974978db3
I20260926 09:36:23.278107 18499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:23.287226 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.275881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.0.98.62:39091)
0926 09:36:23.276062 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.276072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.276107 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:36:23.276282 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.276292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.276713 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.276868 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:23.277530 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.278956 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.280051 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.280639 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.281019 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.283450 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:23.283454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:23.283455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:23.283466 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:23.285400 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.285770 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.285773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.285773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.285867 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.286050 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.286051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.286051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.286330 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:23.286782 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.286849 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.286994 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260926 09:36:23.293208 18499 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:23.293721 18499 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:23.302294 18499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346: Generated new TSK 0
I20260926 09:36:23.302513 18499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:36:23.334707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:23.419585 18518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:23.419793 18518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:23.419831 18518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:23.421068 18518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:23.421129 18518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:23.421175 18518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:36:23.422724 18518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:23.422996 18518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:23.423231 18518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:23.428046 18526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:23.428046 18524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:23.428109 18523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:23.428134 18518 server_base.cc:1034] running on GCE node
I20260926 09:36:23.428550 18518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:23.428800 18518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:23.429976 18518 hybrid_clock.cc:648] HybridClock initialized: now 1790415383429961 us; error 45 us; skew 500 ppm
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:23.432530 18518 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:36:23.432952 18518 webserver.cc:460] Webserver started at http://127.0.98.1:45663/ using document root <none> and password file <none>
I20260926 09:36:23.433139 18518 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:23.433177 18518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:23.433260 18518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:23.434186 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "39f0fba6dba446f480da0600353bf362"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "a6d2595f37e42f7606a534c0e55a376c"
server_key_iv: "1b3a60beb4f80db72145d4f19694df25"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.434453 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "39f0fba6dba446f480da0600353bf362"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "a6d2595f37e42f7606a534c0e55a376c"
server_key_iv: "1b3a60beb4f80db72145d4f19694df25"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.435719 18518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:23.436388 18533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.436585 18518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:23.436648 18518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "39f0fba6dba446f480da0600353bf362"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "a6d2595f37e42f7606a534c0e55a376c"
server_key_iv: "1b3a60beb4f80db72145d4f19694df25"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.436762 18518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:23.464658 18518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:23.467015 18518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:23.467180 18518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:23.467439 18518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:23.467767 18518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:23.467801 18518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.467840 18518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:23.467868 18518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.473600 18518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32897
I20260926 09:36:23.473676 18646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32897 every 8 connection(s)
I20260926 09:36:23.474000 18518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:36:23.479717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 18518
I20260926 09:36:23.479802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:36:23.480052 392 external_mini_cluster.cc:1428] Setting key 8cf873751dce055c2c8f1eeacf701d46
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:23.484032 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.474773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59527 (local address 127.0.98.62:39091)
0926 09:36:23.474823 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.474825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.475027 (+ 202us) server_negotiation.cc:407] Connection header received
0926 09:36:23.476118 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.476119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.476129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.476200 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:23.476818 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.477322 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.478425 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.478637 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.478705 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.480911 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:23.480912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:23.480912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:23.480918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:23.482514 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.482990 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.482991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.482992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.483063 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.483385 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.483386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.483387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.483496 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:23.483556 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.483896 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.483969 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:23.484321 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.474726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.1:59527)
0926 09:36:23.474887 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.474930 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:23.476018 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:23.476272 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:23.476275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:23.476520 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:23.476725 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.476731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.477420 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.477421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.478267 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.478270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.478717 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.478718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.478819 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.479295 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:23.479303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:23.480811 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:23.482622 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:23.482630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:23.482632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:23.482856 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:23.483134 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:23.483135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:23.483136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:23.483278 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:23.483572 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:23.483579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:23.483745 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 09:36:23.483858 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 09:36:23.484044 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260926 09:36:23.484874 18647 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
I20260926 09:36:23.484985 18647 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:23.485185 18647 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260926 09:36:23.485747 18456 ts_manager.cc:194] Registered new tserver with Master: 39f0fba6dba446f480da0600353bf362 (127.0.98.1:32897)
I20260926 09:36:23.486727 18456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59527
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:36:23.505859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:23.584098 18652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:23.584321 18652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:23.584362 18652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:23.585531 18652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:23.585596 18652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:23.585639 18652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:36:23.587066 18652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:23.587342 18652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:23.587558 18652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:23.592258 18658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:23.592263 18660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:23.592276 18657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:23.592536 18652 server_base.cc:1034] running on GCE node
I20260926 09:36:23.592765 18652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:23.592991 18652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:23.594166 18652 hybrid_clock.cc:648] HybridClock initialized: now 1790415383594154 us; error 27 us; skew 500 ppm
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:23.596616 18652 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:36:23.597023 18652 webserver.cc:460] Webserver started at http://127.0.98.2:34031/ using document root <none> and password file <none>
I20260926 09:36:23.597256 18652 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:23.597339 18652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:23.597489 18652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:23.598443 18652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "cba2734e637f42b99a650a234a5ce8ab"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "caf62862dccc58110b2aa9ee619f92d5"
server_key_iv: "b18077fecb1d45f55182b4b43b12cd35"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.598778 18652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cba2734e637f42b99a650a234a5ce8ab"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "caf62862dccc58110b2aa9ee619f92d5"
server_key_iv: "b18077fecb1d45f55182b4b43b12cd35"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.600204 18652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:23.600874 18667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.601034 18652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:23.601130 18652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "cba2734e637f42b99a650a234a5ce8ab"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "caf62862dccc58110b2aa9ee619f92d5"
server_key_iv: "b18077fecb1d45f55182b4b43b12cd35"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.601205 18652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:23.624675 18652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:23.626408 18652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:23.626566 18652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:23.626780 18652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:23.627079 18652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:23.627134 18652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.627173 18652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:23.627200 18652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.632843 18652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43527
I20260926 09:36:23.632886 18780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43527 every 8 connection(s)
I20260926 09:36:23.633172 18652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:23.640214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 18652
I20260926 09:36:23.640339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:36:23.640585 392 external_mini_cluster.cc:1428] Setting key e0dc0248f6e6723b210083c44bb5b8ff
I20260926 09:36:23.642597 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.633766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46591 (local address 127.0.98.62:39091)
0926 09:36:23.633810 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.633812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.634032 (+ 220us) server_negotiation.cc:407] Connection header received
0926 09:36:23.634956 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.634957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.634974 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.635046 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:23.635739 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.636227 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.637221 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.637414 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.637501 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.639539 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:23.639540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:23.639541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:23.639547 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:23.641165 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.641677 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.641678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.641679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.641735 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.642013 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.642014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.642014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.642114 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:23.642164 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.642467 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.642533 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:36:23.642889 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.633750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.2:46591)
0926 09:36:23.633914 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.633946 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:23.634841 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:23.635114 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:23.635118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:23.635449 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:23.635635 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.635639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.636318 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.636319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.637116 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.637118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.637530 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.637531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.637641 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.638058 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:23.638065 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:23.639435 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:23.641230 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:23.641237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:23.641240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:23.641598 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:23.641803 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:23.641804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:23.641804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:23.641950 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:23.642176 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:23.642181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:23.642341 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 09:36:23.642449 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 09:36:23.642624 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260926 09:36:23.643363 18781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
I20260926 09:36:23.643471 18781 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:23.643666 18781 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260926 09:36:23.644032 18456 ts_manager.cc:194] Registered new tserver with Master: cba2734e637f42b99a650a234a5ce8ab (127.0.98.2:43527)
I20260926 09:36:23.644413 18456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46591
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:36:23.665903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:23.746183 18786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:23.746402 18786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:23.746439 18786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:23.747656 18786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:23.747731 18786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:23.747776 18786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:36:23.749241 18786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:23.749490 18786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:23.749711 18786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:23.754537 18794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:23.754552 18792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:23.754717 18786 server_base.cc:1034] running on GCE node
W20260926 09:36:23.754528 18791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:23.754980 18786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:23.755257 18786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:23.756450 18786 hybrid_clock.cc:648] HybridClock initialized: now 1790415383756439 us; error 27 us; skew 500 ppm
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:23.758899 18786 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:36:23.759361 18786 webserver.cc:460] Webserver started at http://127.0.98.3:38699/ using document root <none> and password file <none>
I20260926 09:36:23.759582 18786 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:23.759650 18786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:23.759800 18786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:23.760972 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "242e935d933e4fd3ae1d268179f87ced"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "b6d6e8cad6ef7b9eca486839c3b73a34"
server_key_iv: "d6ed1c7c90bc1beb1f15f2cd682fa2aa"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.761389 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "242e935d933e4fd3ae1d268179f87ced"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "b6d6e8cad6ef7b9eca486839c3b73a34"
server_key_iv: "d6ed1c7c90bc1beb1f15f2cd682fa2aa"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.763058 18786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:23.764055 18801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.764218 18786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:23.764288 18786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "242e935d933e4fd3ae1d268179f87ced"
format_stamp: "Formatted at 2026-09-26 09:36:23 on dist-test-slave-21h8"
server_key: "b6d6e8cad6ef7b9eca486839c3b73a34"
server_key_iv: "d6ed1c7c90bc1beb1f15f2cd682fa2aa"
server_key_version: "encryptionkey@0"
I20260926 09:36:23.764336 18786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:23.792420 18786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:23.794281 18786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:23.794395 18786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:23.794595 18786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:23.794884 18786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:23.794916 18786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.794956 18786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:23.794977 18786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:23.801373 18786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40615
I20260926 09:36:23.801450 18914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40615 every 8 connection(s)
I20260926 09:36:23.801782 18786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:36:23 dist-test-slave-21h8 krb5kdc[18411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:23.810771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 18786
I20260926 09:36:23.810863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:36:23.811060 392 external_mini_cluster.cc:1428] Setting key 9cfcc2e0fcc551b4e0624213e99d101e
I20260926 09:36:23.811175 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.802534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41983 (local address 127.0.98.62:39091)
0926 09:36:23.802596 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.802597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.802842 (+ 245us) server_negotiation.cc:407] Connection header received
0926 09:36:23.804098 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.804100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.804110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.804158 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:23.804712 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.805315 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.806293 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.806501 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.806549 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.808545 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:23.808546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:23.808547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:23.808553 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:23.809910 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.810270 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.810271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.810271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.810321 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.810522 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.810523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.810523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.810617 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:23.810673 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.810998 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.811062 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:36:23.811417 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.802509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.3:41983)
0926 09:36:23.802697 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.802748 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:23.803994 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:23.804210 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:23.804213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:23.804448 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:23.804641 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.804644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.805399 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.805400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.806200 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.806202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.806569 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.806570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.806663 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.807069 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:23.807076 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:23.808465 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:23.809994 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:23.810000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:23.810002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:23.810208 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:23.810376 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:23.810377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:23.810377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:23.810473 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:23.810691 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:23.810697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:23.810864 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 09:36:23.810982 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 09:36:23.811148 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260926 09:36:23.811882 18915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
I20260926 09:36:23.812007 18915 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:23.812203 18915 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260926 09:36:23.812541 18456 ts_manager.cc:194] Registered new tserver with Master: 242e935d933e4fd3ae1d268179f87ced (127.0.98.3:40615)
I20260926 09:36:23.812880 18456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41983
I20260926 09:36:23.822821 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:36:23.829918 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.824232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.0.98.62:39091)
0926 09:36:23.824299 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.824301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.824379 (+ 78us) server_negotiation.cc:407] Connection header received
0926 09:36:23.824420 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.824421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.824430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.824472 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:23.824985 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.825450 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.826134 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.826311 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.826360 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.827654 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:23.827655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:23.827655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:23.827662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:23.829014 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.829327 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.829328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.829329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.829373 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:23.829533 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:23.829534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:23.829534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:23.829641 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:23.829719 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.829811 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.829857 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:36:23.831533 18456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:36:23.833081 18456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:36:23.839717 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.835546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40615 (local address 127.0.0.1:52188)
0926 09:36:23.835751 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.835794 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:23.835860 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:23.836341 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:23.836341 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:23.836343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:23.836381 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.836383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.837714 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.837715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.838455 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.838457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.839394 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.839395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.839484 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.839498 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:36:23.839582 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:36:23.839633 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":57,"threads_started":1}
I20260926 09:36:23.839761 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.835731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.0.98.3:40615)
0926 09:36:23.836020 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.836023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.836032 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:36:23.836071 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.836073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.836090 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.836262 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:23.836467 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.837575 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.838552 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.839290 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.839394 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.839455 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.839610 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.839662 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":91,"threads_started":1}
I20260926 09:36:23.839993 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.835607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32897 (local address 127.0.0.1:59852)
0926 09:36:23.836124 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.836133 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:23.836178 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:23.836489 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:23.836490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:23.836493 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:23.836541 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.836543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.837432 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.837433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.838691 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.838694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.839745 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.839746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.839813 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.839819 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:36:23.839874 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:36:23.839921 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":95,"threads_started":1}
I20260926 09:36:23.840289 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.835858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.0.98.1:32897)
0926 09:36:23.836065 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.836069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.836238 (+ 169us) server_negotiation.cc:407] Connection header received
0926 09:36:23.836285 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.836288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.836305 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.836382 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:23.836613 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.837317 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.838829 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.839587 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.839653 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.839711 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.840150 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.840211 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260926 09:36:23.840693 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.835547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43527 (local address 127.0.0.1:41274)
0926 09:36:23.835938 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.835951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:23.836023 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:23.836306 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:23.836309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:23.836311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:23.836381 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.836383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.837682 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.837683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.839033 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.839036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.840260 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.840262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.840322 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.840327 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:36:23.840577 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 09:36:23.840622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260926 09:36:23.840739 18849 tablet_service.cc:1467] Processing CreateTablet for tablet e2b369946e414c94b052f7b0fffa94f4 (DEFAULT_TABLE table=test-table [id=d711eb9473214ac7ae416d0824193303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:23.840966 18581 tablet_service.cc:1467] Processing CreateTablet for tablet e2b369946e414c94b052f7b0fffa94f4 (DEFAULT_TABLE table=test-table [id=d711eb9473214ac7ae416d0824193303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:23.841169 18849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b369946e414c94b052f7b0fffa94f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:23.841193 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.835739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.0.98.2:43527)
0926 09:36:23.835931 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.835936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.836011 (+ 75us) server_negotiation.cc:407] Connection header received
0926 09:36:23.836104 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.836107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.836123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.836193 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:23.836468 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.837568 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.839146 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.840129 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.840258 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.840320 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.841061 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.841109 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260926 09:36:23.841331 18581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b369946e414c94b052f7b0fffa94f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:23.841854 18715 tablet_service.cc:1467] Processing CreateTablet for tablet e2b369946e414c94b052f7b0fffa94f4 (DEFAULT_TABLE table=test-table [id=d711eb9473214ac7ae416d0824193303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:23.842195 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b369946e414c94b052f7b0fffa94f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:23.844079 18932 tablet_bootstrap.cc:492] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: Bootstrap starting.
I20260926 09:36:23.844695 18932 tablet_bootstrap.cc:654] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:23.844997 18933 tablet_bootstrap.cc:492] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362: Bootstrap starting.
I20260926 09:36:23.845265 18932 log.cc:824] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:23.845534 18934 tablet_bootstrap.cc:492] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Bootstrap starting.
I20260926 09:36:23.845665 18933 tablet_bootstrap.cc:654] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:23.846131 18934 tablet_bootstrap.cc:654] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:23.846158 18933 log.cc:824] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:23.846586 18934 log.cc:824] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:23.846961 18933 tablet_bootstrap.cc:492] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362: No bootstrap required, opened a new log
I20260926 09:36:23.847041 18933 ts_tablet_manager.cc:1397] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:23.847402 18934 tablet_bootstrap.cc:492] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: No bootstrap required, opened a new log
I20260926 09:36:23.847486 18934 ts_tablet_manager.cc:1397] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:23.847498 18932 tablet_bootstrap.cc:492] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: No bootstrap required, opened a new log
I20260926 09:36:23.847558 18932 ts_tablet_manager.cc:1397] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 09:36:23.848834 18932 raft_consensus.cc:348] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.849001 18932 raft_consensus.cc:374] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:23.849045 18932 raft_consensus.cc:729] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242e935d933e4fd3ae1d268179f87ced, State: Initialized, Role: FOLLOWER
I20260926 09:36:23.849182 18932 consensus_queue.cc:260] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.849187 18933 raft_consensus.cc:348] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.849378 18933 raft_consensus.cc:374] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:23.849427 18933 raft_consensus.cc:729] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0fba6dba446f480da0600353bf362, State: Initialized, Role: FOLLOWER
I20260926 09:36:23.849467 18932 ts_tablet_manager.cc:1428] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:23.849483 18934 raft_consensus.cc:348] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.849570 18933 consensus_queue.cc:260] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.849615 18915 heartbeater.cc:503] Master 127.0.98.62:39091 was elected leader, sending a full tablet report...
I20260926 09:36:23.849649 18934 raft_consensus.cc:374] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:23.849704 18934 raft_consensus.cc:729] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba2734e637f42b99a650a234a5ce8ab, State: Initialized, Role: FOLLOWER
I20260926 09:36:23.849849 18933 ts_tablet_manager.cc:1428] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:23.849822 18934 consensus_queue.cc:260] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.849987 18647 heartbeater.cc:503] Master 127.0.98.62:39091 was elected leader, sending a full tablet report...
I20260926 09:36:23.850087 18934 ts_tablet_manager.cc:1428] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:23.850209 18781 heartbeater.cc:503] Master 127.0.98.62:39091 was elected leader, sending a full tablet report...
W20260926 09:36:23.883709 18782 tablet.cc:2369] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:36:23.944767 18938 raft_consensus.cc:482] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:23.944927 18938 raft_consensus.cc:504] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.945322 18938 leader_election.cc:288] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cba2734e637f42b99a650a234a5ce8ab (127.0.98.2:43527), 39f0fba6dba446f480da0600353bf362 (127.0.98.1:32897)
I20260926 09:36:23.949409 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.945585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41327 (local address 127.0.98.1:32897)
0926 09:36:23.945639 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.945641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.945653 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:36:23.945693 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.945694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.945707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.945800 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:23.946003 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.946853 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.948406 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.949029 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.949112 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.949123 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.949291 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.949332 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:36:23.949419 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.945517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32897 (local address 127.0.98.3:41327)
0926 09:36:23.945585 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.945596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:23.945640 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:23.945797 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:23.945798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:23.945800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:23.945885 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.945886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.946939 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.946940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.948310 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.948313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.949118 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.949119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.949207 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.949211 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:23.949268 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:36:23.949316 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":47}
I20260926 09:36:23.949415 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.945626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36043 (local address 127.0.98.2:43527)
0926 09:36:23.945683 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.945686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.945821 (+ 135us) server_negotiation.cc:407] Connection header received
0926 09:36:23.945853 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.945854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.945873 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.945919 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:23.946084 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.946852 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.948406 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.949029 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.949112 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.949123 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.949291 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.949332 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:36:23.949429 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.945558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43527 (local address 127.0.98.3:36043)
0926 09:36:23.945738 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.945746 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:23.945798 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:23.945945 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:23.945946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:23.945947 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:23.946006 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.946008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.946939 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.946940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.948310 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:23.948313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.949119 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.949119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:23.949207 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.949211 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:23.949268 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:36:23.949316 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":110,"thread_start_us":123,"threads_started":1}
I20260926 09:36:23.949805 18601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b369946e414c94b052f7b0fffa94f4" candidate_uuid: "242e935d933e4fd3ae1d268179f87ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0fba6dba446f480da0600353bf362" is_pre_election: true
I20260926 09:36:23.949844 18735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b369946e414c94b052f7b0fffa94f4" candidate_uuid: "242e935d933e4fd3ae1d268179f87ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cba2734e637f42b99a650a234a5ce8ab" is_pre_election: true
I20260926 09:36:23.949972 18601 raft_consensus.cc:2454] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242e935d933e4fd3ae1d268179f87ced in term 0.
I20260926 09:36:23.950011 18735 raft_consensus.cc:2454] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242e935d933e4fd3ae1d268179f87ced in term 0.
I20260926 09:36:23.950214 18805 leader_election.cc:302] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242e935d933e4fd3ae1d268179f87ced, 39f0fba6dba446f480da0600353bf362; no voters:
I20260926 09:36:23.950337 18938 raft_consensus.cc:2781] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:36:23.950389 18938 raft_consensus.cc:482] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:36:23.950423 18938 raft_consensus.cc:3037] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:23.951352 18938 raft_consensus.cc:504] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.951490 18938 leader_election.cc:288] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [CANDIDATE]: Term 1 election: Requested vote from peers cba2734e637f42b99a650a234a5ce8ab (127.0.98.2:43527), 39f0fba6dba446f480da0600353bf362 (127.0.98.1:32897)
I20260926 09:36:23.951674 18601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b369946e414c94b052f7b0fffa94f4" candidate_uuid: "242e935d933e4fd3ae1d268179f87ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0fba6dba446f480da0600353bf362"
I20260926 09:36:23.951682 18735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b369946e414c94b052f7b0fffa94f4" candidate_uuid: "242e935d933e4fd3ae1d268179f87ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cba2734e637f42b99a650a234a5ce8ab"
I20260926 09:36:23.951758 18601 raft_consensus.cc:3037] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:23.951753 18735 raft_consensus.cc:3037] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:23.952425 18601 raft_consensus.cc:2454] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242e935d933e4fd3ae1d268179f87ced in term 1.
I20260926 09:36:23.952589 18735 raft_consensus.cc:2454] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242e935d933e4fd3ae1d268179f87ced in term 1.
I20260926 09:36:23.952661 18805 leader_election.cc:302] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242e935d933e4fd3ae1d268179f87ced, 39f0fba6dba446f480da0600353bf362; no voters:
I20260926 09:36:23.952751 18938 raft_consensus.cc:2781] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:23.952879 18938 raft_consensus.cc:686] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 1 LEADER]: Becoming Leader. State: Replica: 242e935d933e4fd3ae1d268179f87ced, State: Running, Role: LEADER
I20260926 09:36:23.953006 18938 consensus_queue.cc:237] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } }
I20260926 09:36:23.953787 18455 catalog_manager.cc:5495] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 242e935d933e4fd3ae1d268179f87ced (127.0.98.3). New cstate: current_term: 1 leader_uuid: "242e935d933e4fd3ae1d268179f87ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "242e935d933e4fd3ae1d268179f87ced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 } health_report { overall_health: UNKNOWN } } }
W20260926 09:36:23.974821 18648 tablet.cc:2369] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:36:23.984253 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.981966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.0.98.3:40615)
0926 09:36:23.982027 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:23.982029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:23.982038 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:36:23.982081 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:23.982082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:23.982098 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:23.982159 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:23.982323 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.982768 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.983527 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:23.983712 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:23.983763 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:23.983892 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:23.983974 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:23.984062 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:23.984105 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 09:36:23.984112 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:36:23.985986 18601 raft_consensus.cc:1261] T e2b369946e414c94b052f7b0fffa94f4 P 39f0fba6dba446f480da0600353bf362 [term 1 FOLLOWER]: Refusing update from remote peer 242e935d933e4fd3ae1d268179f87ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:23.985988 18735 raft_consensus.cc:1261] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 1 FOLLOWER]: Refusing update from remote peer 242e935d933e4fd3ae1d268179f87ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:23.986334 18942 consensus_queue.cc:1035] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0fba6dba446f480da0600353bf362" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:23.986522 18938 consensus_queue.cc:1035] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "cba2734e637f42b99a650a234a5ce8ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:23.988469 18945 mvcc.cc:205] Tried to move back new op lower bound from 7333541412804030464 to 7333541412672139264. Current Snapshot: MvccSnapshot[applied={T|T < 7333541412804030464}]
I20260926 09:36:23.989451 18946 mvcc.cc:205] Tried to move back new op lower bound from 7333541412804030464 to 7333541412672139264. Current Snapshot: MvccSnapshot[applied={T|T < 7333541412804030464 or (T in {7333541412804030464})}]
I20260926 09:36:23.989557 18947 mvcc.cc:205] Tried to move back new op lower bound from 7333541412804030464 to 7333541412672139264. Current Snapshot: MvccSnapshot[applied={T|T < 7333541412804030464}]
I20260926 09:36:23.990883 18456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:23.990983 18456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:23.991721 18456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ec624a93a1c44a1a90abd82fe810346: Sending DeleteTablet for 3 replicas of tablet e2b369946e414c94b052f7b0fffa94f4
I20260926 09:36:23.992045 18715 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b369946e414c94b052f7b0fffa94f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41274
I20260926 09:36:23.992157 18581 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b369946e414c94b052f7b0fffa94f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59852
I20260926 09:36:23.992268 18849 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b369946e414c94b052f7b0fffa94f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52188
I20260926 09:36:23.992389 18955 tablet_replica.cc:330] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: stopping tablet replica
I20260926 09:36:23.992437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 18518
I20260926 09:36:23.992484 18955 raft_consensus.cc:2229] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:36:23.992585 18955 raft_consensus.cc:2258] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:23.994287 18955 ts_tablet_manager.cc:1905] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:23.996057 18955 ts_tablet_manager.cc:1918] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:36:23.996147 18955 log.cc:1196] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/e2b369946e414c94b052f7b0fffa94f4
I20260926 09:36:23.996447 18955 ts_tablet_manager.cc:1939] T e2b369946e414c94b052f7b0fffa94f4 P cba2734e637f42b99a650a234a5ce8ab: Deleting consensus metadata
I20260926 09:36:23.996871 18441 catalog_manager.cc:4841] TS cba2734e637f42b99a650a234a5ce8ab (127.0.98.2:43527): tablet e2b369946e414c94b052f7b0fffa94f4 (table test-table [id=d711eb9473214ac7ae416d0824193303]) successfully deleted
W20260926 09:36:23.998399 18443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32897 (error 108)
I20260926 09:36:23.998582 18957 tablet_replica.cc:330] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: stopping tablet replica
I20260926 09:36:23.998716 18957 raft_consensus.cc:2229] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:36:23.998839 18957 raft_consensus.cc:2258] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:23.998980 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:23.998827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32897 (local address 127.0.0.1:59862)
0926 09:36:23.998899 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:23.998941 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 09:36:23.999033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 18652
W20260926 09:36:23.999102 18443 catalog_manager.cc:4584] TS 39f0fba6dba446f480da0600353bf362 (127.0.98.1:32897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2b369946e414c94b052f7b0fffa94f4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32897: connect: Connection refused (error 111)
I20260926 09:36:24.000047 18957 ts_tablet_manager.cc:1905] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:24.001972 18957 ts_tablet_manager.cc:1918] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:36:24.002045 18957 log.cc:1196] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/e2b369946e414c94b052f7b0fffa94f4
I20260926 09:36:24.002318 18957 ts_tablet_manager.cc:1939] T e2b369946e414c94b052f7b0fffa94f4 P 242e935d933e4fd3ae1d268179f87ced: Deleting consensus metadata
I20260926 09:36:24.002722 18440 catalog_manager.cc:4841] TS 242e935d933e4fd3ae1d268179f87ced (127.0.98.3:40615): tablet e2b369946e414c94b052f7b0fffa94f4 (table test-table [id=d711eb9473214ac7ae416d0824193303]) successfully deleted
I20260926 09:36:24.005301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 18786
I20260926 09:36:24.010113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 18424
2026-09-26T09:36:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2048 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T09:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:24Z Disabled control of system clock
I20260926 09:36:24.037302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33163
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:36:24.115679 18962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:24.115877 18962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:24.115897 18962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:24.116981 18962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:24.117013 18962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:24.117026 18962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:24.117039 18962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:24.117050 18962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:24.118397 18962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33163
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:24.118579 18962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:24.118800 18962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:24.123438 18968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:24.123426 18967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:24.123402 18970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:24.123522 18962 server_base.cc:1034] running on GCE node
I20260926 09:36:24.123910 18962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:24.124194 18962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:24.125332 18962 hybrid_clock.cc:648] HybridClock initialized: now 1790415384125312 us; error 36 us; skew 500 ppm
I20260926 09:36:24.126569 18962 webserver.cc:460] Webserver started at http://127.0.98.62:33383/ using document root <none> and password file <none>
I20260926 09:36:24.126827 18962 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:24.126875 18962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:24.127031 18962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:24.128093 18962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "3467a618e9d941839ec7606e8661465d"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "793bfb9c33df16d6a17f706c971afb70"
server_key_iv: "18c5e87d66f01299e039ab837e14f3fe"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.128432 18962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "3467a618e9d941839ec7606e8661465d"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "793bfb9c33df16d6a17f706c971afb70"
server_key_iv: "18c5e87d66f01299e039ab837e14f3fe"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.129637 18962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:36:24.130308 18976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.130463 18962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:24.130573 18962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "3467a618e9d941839ec7606e8661465d"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "793bfb9c33df16d6a17f706c971afb70"
server_key_iv: "18c5e87d66f01299e039ab837e14f3fe"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.130654 18962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:24.144310 18962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:24.144585 18962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:24.144742 18962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:24.148579 18962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33163
I20260926 09:36:24.148619 19028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33163 every 8 connection(s)
I20260926 09:36:24.148928 18962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:24.149821 19029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:24.151232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 18962
I20260926 09:36:24.151355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:24.151575 392 external_mini_cluster.cc:1428] Setting key 5311d1b619f53cfc8b555a46bd30d15a
I20260926 09:36:24.152370 19029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d: Bootstrap starting.
I20260926 09:36:24.152969 19029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:24.153298 19029 log.cc:824] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:24.153951 19029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d: No bootstrap required, opened a new log
I20260926 09:36:24.156409 19029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3467a618e9d941839ec7606e8661465d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33163 } }
I20260926 09:36:24.156562 19029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:24.156586 19029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3467a618e9d941839ec7606e8661465d, State: Initialized, Role: FOLLOWER
I20260926 09:36:24.156713 19029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3467a618e9d941839ec7606e8661465d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33163 } }
I20260926 09:36:24.156790 19029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:24.156828 19029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:24.156875 19029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:24.157088 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.152075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.0.98.62:33163)
0926 09:36:24.152274 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.152281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.152305 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:36:24.152757 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.152764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.152823 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.152940 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:24.153203 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.154106 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.155188 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.155625 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.155720 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.156425 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:24.156428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:24.156428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:24.156619 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:24.156666 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:24.156678 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:24.156764 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.156841 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.156923 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260926 09:36:24.157505 19029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3467a618e9d941839ec7606e8661465d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33163 } }
I20260926 09:36:24.157604 19029 leader_election.cc:302] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3467a618e9d941839ec7606e8661465d; no voters:
I20260926 09:36:24.157756 19029 leader_election.cc:288] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:24.157789 19034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:24.157935 19034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [term 1 LEADER]: Becoming Leader. State: Replica: 3467a618e9d941839ec7606e8661465d, State: Running, Role: LEADER
I20260926 09:36:24.157997 19029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:24.158021 19034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3467a618e9d941839ec7606e8661465d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33163 } }
I20260926 09:36:24.158376 19035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3467a618e9d941839ec7606e8661465d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3467a618e9d941839ec7606e8661465d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33163 } } }
I20260926 09:36:24.158375 19036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3467a618e9d941839ec7606e8661465d. Latest consensus state: current_term: 1 leader_uuid: "3467a618e9d941839ec7606e8661465d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3467a618e9d941839ec7606e8661465d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33163 } } }
I20260926 09:36:24.158560 19035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:24.158576 19036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:24.158910 19044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:24.159453 19044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:24.160918 19044 catalog_manager.cc:1347] Generated new cluster ID: b60df23e0f784d60b12572b3d5c0a66e
I20260926 09:36:24.160980 19044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:24.174917 19044 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:24.175640 19044 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:24.190826 19044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d: Generated new TSK 0
I20260926 09:36:24.191025 19044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:24.199523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33163
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:36:24.279366 19053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:24.279582 19053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:24.279618 19053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:24.280829 19053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:24.280893 19053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:24.280937 19053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:36:24.282280 19053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:24.282529 19053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:24.282819 19053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:24.288028 19058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:24.288098 19053 server_base.cc:1034] running on GCE node
W20260926 09:36:24.288045 19061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:24.288028 19059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:24.288419 19053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:24.288681 19053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:24.289839 19053 hybrid_clock.cc:648] HybridClock initialized: now 1790415384289820 us; error 31 us; skew 500 ppm
I20260926 09:36:24.296990 19053 webserver.cc:460] Webserver started at http://127.0.98.1:46455/ using document root <none> and password file <none>
I20260926 09:36:24.297252 19053 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:24.297330 19053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:24.297503 19053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:24.298604 19053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "d6a01403428c47b9ab2834e8b2979f9f"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "6ed3cda7ec8fb99273b3e6265adf6446"
server_key_iv: "9a79e997a8d46280245669857ac79347"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.298954 19053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d6a01403428c47b9ab2834e8b2979f9f"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "6ed3cda7ec8fb99273b3e6265adf6446"
server_key_iv: "9a79e997a8d46280245669857ac79347"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.300324 19053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:24.301047 19067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.301236 19053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:24.301312 19053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "d6a01403428c47b9ab2834e8b2979f9f"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "6ed3cda7ec8fb99273b3e6265adf6446"
server_key_iv: "9a79e997a8d46280245669857ac79347"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.301394 19053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:24.313853 19053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:24.314142 19053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:24.314311 19053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:24.314519 19053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:24.314817 19053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:24.314870 19053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.314910 19053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:24.314939 19053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.320750 19053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38161
I20260926 09:36:24.320787 19180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38161 every 8 connection(s)
I20260926 09:36:24.321130 19053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:36:24.324389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 19053
I20260926 09:36:24.324489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:36:24.324685 392 external_mini_cluster.cc:1428] Setting key 44f9e78dc6a593b85999cc0c70f54e6c
I20260926 09:36:24.325543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33163
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:36:24.326535 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.321815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58527 (local address 127.0.98.62:33163)
0926 09:36:24.321859 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.321863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.322061 (+ 198us) server_negotiation.cc:407] Connection header received
0926 09:36:24.322579 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.322581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.322593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.322649 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:24.323345 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.323843 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.325135 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.325335 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.325421 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.326273 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:24.326274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:24.326274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:24.326279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:24.326294 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:24.326298 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:24.326364 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.326435 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.326461 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:36:24.326855 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.321778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33163 (local address 127.0.98.1:58527)
0926 09:36:24.321948 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.321978 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:24.322481 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:24.322720 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:24.322724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:24.322733 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:24.323232 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.323237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.323941 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.323943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.325043 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.325045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.325430 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.325431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.325558 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.326118 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:24.326118 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:24.326145 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:24.326167 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:24.326365 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:24.326372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:24.326375 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:24.326441 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:36:24.326551 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260926 09:36:24.327296 19181 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33163
I20260926 09:36:24.327417 19181 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:24.327600 19181 heartbeater.cc:511] Master 127.0.98.62:33163 requested a full tablet report, sending...
I20260926 09:36:24.328095 18993 ts_manager.cc:194] Registered new tserver with Master: d6a01403428c47b9ab2834e8b2979f9f (127.0.98.1:38161)
I20260926 09:36:24.328691 18993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58527
W20260926 09:36:24.405051 19184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:24.405264 19184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:24.405300 19184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:24.406459 19184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:24.406523 19184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:24.406567 19184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:36:24.408034 19184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:24.408279 19184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:24.408553 19184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:24.413357 19192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:24.413388 19190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:24.413401 19189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:24.413700 19184 server_base.cc:1034] running on GCE node
I20260926 09:36:24.413892 19184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:24.414086 19184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:24.415230 19184 hybrid_clock.cc:648] HybridClock initialized: now 1790415384415207 us; error 38 us; skew 500 ppm
I20260926 09:36:24.416520 19184 webserver.cc:460] Webserver started at http://127.0.98.2:39411/ using document root <none> and password file <none>
I20260926 09:36:24.416718 19184 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:24.416759 19184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:24.416846 19184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:24.417824 19184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "287cce687e8249ebb0c959f9a3293d9e"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "3c8c454cd6d757426b8e0025ce1b9e11"
server_key_iv: "488c69bdfdc22fdf39c4564b0579e5f1"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.418104 19184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "287cce687e8249ebb0c959f9a3293d9e"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "3c8c454cd6d757426b8e0025ce1b9e11"
server_key_iv: "488c69bdfdc22fdf39c4564b0579e5f1"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.419274 19184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:24.419982 19198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.420213 19184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:24.420284 19184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "287cce687e8249ebb0c959f9a3293d9e"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "3c8c454cd6d757426b8e0025ce1b9e11"
server_key_iv: "488c69bdfdc22fdf39c4564b0579e5f1"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.420336 19184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:24.440310 19184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:24.440537 19184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:24.440630 19184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:24.440780 19184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:24.441013 19184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:24.441042 19184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.441062 19184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:24.441074 19184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.446574 19184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33043
I20260926 09:36:24.446645 19311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33043 every 8 connection(s)
I20260926 09:36:24.446944 19184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:36:24.450470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 19184
I20260926 09:36:24.450580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:36:24.450773 392 external_mini_cluster.cc:1428] Setting key 16a66f66fcfd7d6841a42a0fe431b43b
I20260926 09:36:24.451809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33163
--builtin_ntp_servers=127.0.98.20:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:36:24.452725 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.447742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42131 (local address 127.0.98.62:33163)
0926 09:36:24.447846 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.447849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.448051 (+ 202us) server_negotiation.cc:407] Connection header received
0926 09:36:24.448587 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.448588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.448601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.448668 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:24.449381 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.449894 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.451274 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.451581 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.451674 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.452419 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:24.452421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:24.452421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:24.452426 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:24.452443 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:24.452448 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:24.452507 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.452570 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.452598 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 09:36:24.452800 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.447650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33163 (local address 127.0.98.2:42131)
0926 09:36:24.447915 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.447967 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:24.448479 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:24.448733 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:24.448737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:24.448746 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:24.449277 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.449282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.450144 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.450146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.451198 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.451200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.451685 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.451686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.451790 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.452280 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:24.452281 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:24.452307 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:24.452327 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:24.452513 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:24.452516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:24.452518 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:36:24.452568 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:36:24.452638 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":120,"threads_started":1}
I20260926 09:36:24.453269 19312 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33163
I20260926 09:36:24.453369 19312 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:24.453554 19312 heartbeater.cc:511] Master 127.0.98.62:33163 requested a full tablet report, sending...
I20260926 09:36:24.453951 18993 ts_manager.cc:194] Registered new tserver with Master: 287cce687e8249ebb0c959f9a3293d9e (127.0.98.2:33043)
I20260926 09:36:24.454476 18993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42131
W20260926 09:36:24.532333 19315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:24.532531 19315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:24.532553 19315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:24.533680 19315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:24.533720 19315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:24.533744 19315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:36:24.535095 19315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:24.535351 19315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:24.535645 19315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:24.540510 19321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:24.540527 19323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:24.540572 19315 server_base.cc:1034] running on GCE node
W20260926 09:36:24.540527 19320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:24.540911 19315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:24.541107 19315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:24.542281 19315 hybrid_clock.cc:648] HybridClock initialized: now 1790415384542258 us; error 45 us; skew 500 ppm
I20260926 09:36:24.543617 19315 webserver.cc:460] Webserver started at http://127.0.98.3:33703/ using document root <none> and password file <none>
I20260926 09:36:24.543859 19315 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:24.543931 19315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:24.544088 19315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:24.545125 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "d580a142999344b387dab2afd0cfc322"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "45c5acdac259b04a0fd2145b61347192"
server_key_iv: "b38c1b3b76b03f4a6330e6163f44bc26"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.545487 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d580a142999344b387dab2afd0cfc322"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "45c5acdac259b04a0fd2145b61347192"
server_key_iv: "b38c1b3b76b03f4a6330e6163f44bc26"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.546770 19315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:24.547545 19329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.547711 19315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:24.547808 19315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "d580a142999344b387dab2afd0cfc322"
format_stamp: "Formatted at 2026-09-26 09:36:24 on dist-test-slave-21h8"
server_key: "45c5acdac259b04a0fd2145b61347192"
server_key_iv: "b38c1b3b76b03f4a6330e6163f44bc26"
server_key_version: "encryptionkey@0"
I20260926 09:36:24.547888 19315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:24.555996 19315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:24.556254 19315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:24.556396 19315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:24.556599 19315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:24.556905 19315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:24.556957 19315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.556993 19315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:24.557022 19315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:24.562929 19315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44113
I20260926 09:36:24.562980 19442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44113 every 8 connection(s)
I20260926 09:36:24.563277 19315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:36:24.567492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 19315
I20260926 09:36:24.567619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:36:24.567821 392 external_mini_cluster.cc:1428] Setting key 6fef86f0e8739a6025f83e714b1e5bb8
I20260926 09:36:24.568470 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.564042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51395 (local address 127.0.98.62:33163)
0926 09:36:24.564102 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.564104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.564311 (+ 207us) server_negotiation.cc:407] Connection header received
0926 09:36:24.564745 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.564746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.564755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.564816 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:24.565457 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.565964 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.566894 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.567100 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.567167 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.568060 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:24.568062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:24.568062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:24.568068 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:24.568086 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:24.568092 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:24.568155 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.568315 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.568344 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:36:24.568635 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.564018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33163 (local address 127.0.98.3:51395)
0926 09:36:24.564192 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.564231 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:24.564670 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:24.564866 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:24.564869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:24.564877 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:24.565338 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.565342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.566049 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.566049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.566803 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.566805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.567188 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.567189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.567291 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.567915 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:24.567916 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:24.567954 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:24.567979 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:24.568218 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:24.568224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:24.568228 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:24.568298 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:36:24.568421 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260926 09:36:24.569024 19443 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33163
I20260926 09:36:24.569132 19443 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:24.569317 19443 heartbeater.cc:511] Master 127.0.98.62:33163 requested a full tablet report, sending...
I20260926 09:36:24.569650 18993 ts_manager.cc:194] Registered new tserver with Master: d580a142999344b387dab2afd0cfc322 (127.0.98.3:44113)
I20260926 09:36:24.569994 18993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51395
I20260926 09:36:24.580018 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:36:24.584446 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.581449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.0.98.62:33163)
0926 09:36:24.581523 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.581525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.581623 (+ 98us) server_negotiation.cc:407] Connection header received
0926 09:36:24.581707 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.581708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.581719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.581765 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:24.581956 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.582427 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.583183 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.583400 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.583469 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.584151 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:24.584152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:24.584153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:24.584157 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:24.584170 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:24.584174 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:24.584286 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.584354 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.584364 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 09:36:24.586122 18993 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:36:24.587760 18993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:36:24.594120 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.590329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.0.98.3:44113)
0926 09:36:24.590491 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.590495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.590634 (+ 139us) server_negotiation.cc:407] Connection header received
0926 09:36:24.590688 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.590690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.590707 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.590777 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:24.590966 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.591747 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.593239 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.593714 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.593812 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.593899 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.593991 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.594029 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260926 09:36:24.594168 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.589875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.0.1:50708)
0926 09:36:24.590029 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.590038 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:24.590078 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:24.590501 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:24.590504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:24.590506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:24.590608 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.590610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.592060 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.592061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.592924 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.592927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.593882 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.593883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.593954 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.593959 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:36:24.593992 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 09:36:24.594038 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":5,"thread_start_us":120,"threads_started":1}
I20260926 09:36:24.594280 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.589851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33043 (local address 127.0.0.1:34366)
0926 09:36:24.589961 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.590000 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:24.590071 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:24.590501 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:24.590504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:24.590510 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:24.590612 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.590615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.592016 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.592017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.593140 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.593142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.593964 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.593965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.594019 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.594022 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:24.594050 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 09:36:24.594083 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":86,"thread_start_us":66,"threads_started":1}
I20260926 09:36:24.594154 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.589877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44113 (local address 127.0.0.1:38100)
0926 09:36:24.590495 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.590505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:24.590567 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:24.590842 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:24.590843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:24.590845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:24.590878 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.590879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.591839 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.591840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.593143 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.593145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.593811 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.593812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.593884 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.593896 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 09:36:24.593978 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:36:24.594032 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":35,"thread_start_us":87,"threads_started":1}
I20260926 09:36:24.594489 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.590113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.0.98.1:38161)
0926 09:36:24.590308 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.590314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.590324 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:24.590367 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.590369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.590385 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.590775 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:24.590839 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.591960 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.593008 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.593777 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.594213 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.594320 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.594348 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.594396 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260926 09:36:24.595116 19115 tablet_service.cc:1467] Processing CreateTablet for tablet 0e21f07f68e340c3b46cecf3fe65d0e0 (DEFAULT_TABLE table=test-table [id=6b7865aa480b4a05b0a444bbc220fa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:24.595197 19377 tablet_service.cc:1467] Processing CreateTablet for tablet 0e21f07f68e340c3b46cecf3fe65d0e0 (DEFAULT_TABLE table=test-table [id=6b7865aa480b4a05b0a444bbc220fa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:24.595527 19115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e21f07f68e340c3b46cecf3fe65d0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:24.595556 19377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e21f07f68e340c3b46cecf3fe65d0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:24.597949 19457 tablet_bootstrap.cc:492] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: Bootstrap starting.
I20260926 09:36:24.598107 19458 tablet_bootstrap.cc:492] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f: Bootstrap starting.
I20260926 09:36:24.598346 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.590144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.0.98.2:33043)
0926 09:36:24.590330 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.590334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.590344 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:24.590378 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.590380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.590393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.590538 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:24.590710 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.591931 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.593212 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.593885 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.594546 (+ 661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.594634 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.594667 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.594721 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260926 09:36:24.598759 19457 tablet_bootstrap.cc:654] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:24.598770 19458 tablet_bootstrap.cc:654] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:24.599058 19457 log.cc:824] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:24.599071 19458 log.cc:824] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:24.599000 19246 tablet_service.cc:1467] Processing CreateTablet for tablet 0e21f07f68e340c3b46cecf3fe65d0e0 (DEFAULT_TABLE table=test-table [id=6b7865aa480b4a05b0a444bbc220fa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:24.599354 19246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e21f07f68e340c3b46cecf3fe65d0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:24.599792 19457 tablet_bootstrap.cc:492] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: No bootstrap required, opened a new log
I20260926 09:36:24.599861 19457 ts_tablet_manager.cc:1397] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:36:24.601161 19458 tablet_bootstrap.cc:492] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f: No bootstrap required, opened a new log
I20260926 09:36:24.601244 19458 ts_tablet_manager.cc:1397] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:36:24.602391 19457 raft_consensus.cc:348] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.602564 19457 raft_consensus.cc:374] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:24.602600 19457 raft_consensus.cc:729] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d580a142999344b387dab2afd0cfc322, State: Initialized, Role: FOLLOWER
I20260926 09:36:24.602711 19457 consensus_queue.cc:260] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.602831 19461 tablet_bootstrap.cc:492] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e: Bootstrap starting.
I20260926 09:36:24.602972 19457 ts_tablet_manager.cc:1428] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:24.603103 19443 heartbeater.cc:503] Master 127.0.98.62:33163 was elected leader, sending a full tablet report...
I20260926 09:36:24.603469 19461 tablet_bootstrap.cc:654] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:24.603425 19458 raft_consensus.cc:348] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.603588 19458 raft_consensus.cc:374] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:24.603636 19458 raft_consensus.cc:729] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a01403428c47b9ab2834e8b2979f9f, State: Initialized, Role: FOLLOWER
I20260926 09:36:24.603790 19461 log.cc:824] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:24.603788 19458 consensus_queue.cc:260] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.604049 19458 ts_tablet_manager.cc:1428] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:24.604240 19181 heartbeater.cc:503] Master 127.0.98.62:33163 was elected leader, sending a full tablet report...
I20260926 09:36:24.604534 19461 tablet_bootstrap.cc:492] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e: No bootstrap required, opened a new log
I20260926 09:36:24.604604 19461 ts_tablet_manager.cc:1397] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:36:24.606410 19461 raft_consensus.cc:348] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.606526 19461 raft_consensus.cc:374] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:24.606547 19461 raft_consensus.cc:729] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287cce687e8249ebb0c959f9a3293d9e, State: Initialized, Role: FOLLOWER
I20260926 09:36:24.606678 19461 consensus_queue.cc:260] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.606911 19461 ts_tablet_manager.cc:1428] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:36:24.606979 19312 heartbeater.cc:503] Master 127.0.98.62:33163 was elected leader, sending a full tablet report...
I20260926 09:36:24.652189 19465 raft_consensus.cc:482] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:24.652302 19465 raft_consensus.cc:504] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.652621 19465 leader_election.cc:288] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d580a142999344b387dab2afd0cfc322 (127.0.98.3:44113), d6a01403428c47b9ab2834e8b2979f9f (127.0.98.1:38161)
I20260926 09:36:24.655557 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.652766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.98.2:44749)
0926 09:36:24.652813 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.652826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:24.652871 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:24.653028 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:24.653028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:24.653029 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:24.653227 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.653229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.654028 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.654029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.654752 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.654754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.655304 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.655305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.655400 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.655405 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:36:24.655457 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:36:24.655502 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260926 09:36:24.655637 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.652822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44749 (local address 127.0.98.1:38161)
0926 09:36:24.652868 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.652870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.652882 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:36:24.652920 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.652921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.652934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.653003 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:24.653304 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.653930 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.654843 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.655221 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.655275 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.655282 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.655480 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.655523 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:36:24.655714 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.652766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44113 (local address 127.0.98.2:44799)
0926 09:36:24.652992 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.653002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:24.653064 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:24.653302 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:24.653303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:24.653305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:24.653371 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.653372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.653988 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.653989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.654814 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:24.654817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.655418 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.655419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:24.655526 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.655530 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:24.655584 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:36:24.655619 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":43,"thread_start_us":110,"threads_started":1}
I20260926 09:36:24.655748 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.652838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44799 (local address 127.0.98.3:44113)
0926 09:36:24.652876 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.652878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.653172 (+ 294us) server_negotiation.cc:407] Connection header received
0926 09:36:24.653208 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.653209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.653218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.653281 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:24.653440 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.653884 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.654908 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.655280 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.655391 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.655401 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.655609 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.655668 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:36:24.655941 19135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e21f07f68e340c3b46cecf3fe65d0e0" candidate_uuid: "287cce687e8249ebb0c959f9a3293d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a01403428c47b9ab2834e8b2979f9f" is_pre_election: true
I20260926 09:36:24.656047 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e21f07f68e340c3b46cecf3fe65d0e0" candidate_uuid: "287cce687e8249ebb0c959f9a3293d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d580a142999344b387dab2afd0cfc322" is_pre_election: true
I20260926 09:36:24.656106 19135 raft_consensus.cc:2454] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287cce687e8249ebb0c959f9a3293d9e in term 0.
I20260926 09:36:24.656179 19397 raft_consensus.cc:2454] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287cce687e8249ebb0c959f9a3293d9e in term 0.
I20260926 09:36:24.656329 19202 leader_election.cc:302] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287cce687e8249ebb0c959f9a3293d9e, d6a01403428c47b9ab2834e8b2979f9f; no voters:
I20260926 09:36:24.656484 19465 raft_consensus.cc:2781] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:36:24.656538 19465 raft_consensus.cc:482] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:36:24.656574 19465 raft_consensus.cc:3037] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:24.657260 19465 raft_consensus.cc:504] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.657402 19465 leader_election.cc:288] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [CANDIDATE]: Term 1 election: Requested vote from peers d580a142999344b387dab2afd0cfc322 (127.0.98.3:44113), d6a01403428c47b9ab2834e8b2979f9f (127.0.98.1:38161)
I20260926 09:36:24.657531 19135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e21f07f68e340c3b46cecf3fe65d0e0" candidate_uuid: "287cce687e8249ebb0c959f9a3293d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a01403428c47b9ab2834e8b2979f9f"
I20260926 09:36:24.657552 19397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e21f07f68e340c3b46cecf3fe65d0e0" candidate_uuid: "287cce687e8249ebb0c959f9a3293d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d580a142999344b387dab2afd0cfc322"
I20260926 09:36:24.657615 19397 raft_consensus.cc:3037] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:24.657605 19135 raft_consensus.cc:3037] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:24.658529 19397 raft_consensus.cc:2454] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287cce687e8249ebb0c959f9a3293d9e in term 1.
I20260926 09:36:24.658529 19135 raft_consensus.cc:2454] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287cce687e8249ebb0c959f9a3293d9e in term 1.
I20260926 09:36:24.658702 19200 leader_election.cc:302] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287cce687e8249ebb0c959f9a3293d9e, d580a142999344b387dab2afd0cfc322; no voters:
I20260926 09:36:24.658780 19465 raft_consensus.cc:2781] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:24.658931 19465 raft_consensus.cc:686] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 1 LEADER]: Becoming Leader. State: Replica: 287cce687e8249ebb0c959f9a3293d9e, State: Running, Role: LEADER
I20260926 09:36:24.659055 19465 consensus_queue.cc:237] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260926 09:36:24.659920 18993 catalog_manager.cc:5495] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 287cce687e8249ebb0c959f9a3293d9e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "287cce687e8249ebb0c959f9a3293d9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287cce687e8249ebb0c959f9a3293d9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } health_report { overall_health: UNKNOWN } } }
I20260926 09:36:24.668860 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.666769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.0.98.2:33043)
0926 09:36:24.666806 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:24.666807 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:24.666816 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:36:24.666873 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:24.666874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:24.666888 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:24.666947 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:24.667063 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.667555 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.668261 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:24.668431 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:24.668477 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:24.668594 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:24.668679 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:24.668744 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:24.668792 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 09:36:24.668799 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:36:24.670441 19397 raft_consensus.cc:1261] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 1 FOLLOWER]: Refusing update from remote peer 287cce687e8249ebb0c959f9a3293d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:24.670464 19135 raft_consensus.cc:1261] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f [term 1 FOLLOWER]: Refusing update from remote peer 287cce687e8249ebb0c959f9a3293d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:24.670799 19465 consensus_queue.cc:1035] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d580a142999344b387dab2afd0cfc322" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:24.670908 19467 consensus_queue.cc:1035] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a01403428c47b9ab2834e8b2979f9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:24.673492 19470 mvcc.cc:205] Tried to move back new op lower bound from 7333541415607767040 to 7333541415564210176. Current Snapshot: MvccSnapshot[applied={T|T < 7333541415607767040 or (T in {7333541415607767040})}]
I20260926 09:36:24.673753 19472 mvcc.cc:205] Tried to move back new op lower bound from 7333541415607767040 to 7333541415564210176. Current Snapshot: MvccSnapshot[applied={T|T < 7333541415607767040}]
I20260926 09:36:24.674014 19471 mvcc.cc:205] Tried to move back new op lower bound from 7333541415607767040 to 7333541415564210176. Current Snapshot: MvccSnapshot[applied={T|T < 7333541415607767040}]
I20260926 09:36:24.675510 18992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:24.675596 18992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:24.676335 18992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3467a618e9d941839ec7606e8661465d: Sending DeleteTablet for 3 replicas of tablet 0e21f07f68e340c3b46cecf3fe65d0e0
I20260926 09:36:24.676659 19377 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e21f07f68e340c3b46cecf3fe65d0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:24 UTC) from {username='slave'} at 127.0.0.1:38100
I20260926 09:36:24.676699 19115 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e21f07f68e340c3b46cecf3fe65d0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:24 UTC) from {username='slave'} at 127.0.0.1:50708
I20260926 09:36:24.676930 19480 tablet_replica.cc:330] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d6a01403428c47b9ab2834e8b2979f9f: stopping tablet replica
I20260926 09:36:24.677073 19246 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e21f07f68e340c3b46cecf3fe65d0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:24 UTC) from {username='slave'} at 127.0.0.1:34366
I20260926 09:36:24.677153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 19053
I20260926 09:36:24.682952 19481 tablet_replica.cc:330] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e: stopping tablet replica
W20260926 09:36:24.682945 18980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38161 (error 108)
I20260926 09:36:24.683095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 19184
I20260926 09:36:24.683164 19481 raft_consensus.cc:2229] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:36:24.683279 19481 raft_consensus.cc:2258] T 0e21f07f68e340c3b46cecf3fe65d0e0 P 287cce687e8249ebb0c959f9a3293d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:24.683727 19479 tablet_replica.cc:330] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: stopping tablet replica
I20260926 09:36:24.683866 19479 raft_consensus.cc:2229] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:36:24.683982 19479 raft_consensus.cc:2258] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:24.684502 19479 ts_tablet_manager.cc:1905] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:24.685993 19479 ts_tablet_manager.cc:1918] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:36:24.686077 19479 log.cc:1196] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/0e21f07f68e340c3b46cecf3fe65d0e0
I20260926 09:36:24.686363 19479 ts_tablet_manager.cc:1939] T 0e21f07f68e340c3b46cecf3fe65d0e0 P d580a142999344b387dab2afd0cfc322: Deleting consensus metadata
I20260926 09:36:24.686367 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.686216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.0.1:50714)
0926 09:36:24.686278 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.686327 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 09:36:24.686494 18980 catalog_manager.cc:4584] TS d6a01403428c47b9ab2834e8b2979f9f (127.0.98.1:38161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e21f07f68e340c3b46cecf3fe65d0e0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38161: connect: Connection refused (error 111)
I20260926 09:36:24.686803 18978 catalog_manager.cc:4841] TS d580a142999344b387dab2afd0cfc322 (127.0.98.3:44113): tablet 0e21f07f68e340c3b46cecf3fe65d0e0 (table test-table [id=6b7865aa480b4a05b0a444bbc220fa8c]) successfully deleted
I20260926 09:36:24.688349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 19315
I20260926 09:36:24.688393 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:24.688262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33043 (local address 127.0.0.1:34386)
0926 09:36:24.688341 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:24.688368 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 09:36:24.693387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 18962
2026-09-26T09:36:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (682 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:24 dist-test-slave-21h8 krb5kdc[19485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:24 dist-test-slave-21h8 krb5kdc[19485](info): set up 2 sockets
Sep 26 09:36:24 dist-test-slave-21h8 krb5kdc[19485](info): commencing operation
krb5kdc: starting...
W20260926 09:36:25.731153 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:25 dist-test-slave-21h8 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:25.853781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:25.933640 19498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:25.933845 19498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:25.933867 19498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:25.935010 19498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:25.935055 19498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:25.935068 19498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:25.935081 19498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:25.935130 19498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:25.936640 19498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:25.936897 19498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:25.937132 19498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:25.941846 19503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:25.941900 19498 server_base.cc:1034] running on GCE node
W20260926 09:36:25.941881 19506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:25.941855 19504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:25.942307 19498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:25.942592 19498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:25.943734 19498 hybrid_clock.cc:648] HybridClock initialized: now 1790415385943719 us; error 40 us; skew 500 ppm
Sep 26 09:36:25 dist-test-slave-21h8 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:25.946156 19498 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:25.946537 19498 webserver.cc:460] Webserver started at http://127.0.98.62:32863/ using document root <none> and password file <none>
I20260926 09:36:25.946729 19498 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:25.946768 19498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:25.946854 19498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:25.947921 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "811e0bd9b5de4d6895a1937786317ca9"
format_stamp: "Formatted at 2026-09-26 09:36:25 on dist-test-slave-21h8"
server_key: "4904d8536eafdb879d217b4e8cdeba36"
server_key_iv: "faa51c8e0b0f0cda9ae4312c6d0b6113"
server_key_version: "encryptionkey@0"
I20260926 09:36:25.948211 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "811e0bd9b5de4d6895a1937786317ca9"
format_stamp: "Formatted at 2026-09-26 09:36:25 on dist-test-slave-21h8"
server_key: "4904d8536eafdb879d217b4e8cdeba36"
server_key_iv: "faa51c8e0b0f0cda9ae4312c6d0b6113"
server_key_version: "encryptionkey@0"
I20260926 09:36:25.949396 19498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:36:25.950037 19513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:25.950244 19498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:25.950343 19498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "811e0bd9b5de4d6895a1937786317ca9"
format_stamp: "Formatted at 2026-09-26 09:36:25 on dist-test-slave-21h8"
server_key: "4904d8536eafdb879d217b4e8cdeba36"
server_key_iv: "faa51c8e0b0f0cda9ae4312c6d0b6113"
server_key_version: "encryptionkey@0"
I20260926 09:36:25.950420 19498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:25.967972 19498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:25.969774 19498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:25.969933 19498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:25.973912 19498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43603
I20260926 09:36:25.973946 19565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43603 every 8 connection(s)
I20260926 09:36:25.974299 19498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:25.975211 19566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:25.977418 19566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9: Bootstrap starting.
I20260926 09:36:25.977928 19566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:25.978159 19566 log.cc:824] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:25.978434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 19498
I20260926 09:36:25.978562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:25.978791 392 external_mini_cluster.cc:1428] Setting key 632ef2794485f1adb70b5164a6f4901c
I20260926 09:36:25.979025 19566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9: No bootstrap required, opened a new log
I20260926 09:36:25.981222 19566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811e0bd9b5de4d6895a1937786317ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } }
I20260926 09:36:25.981397 19566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:25.981436 19566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811e0bd9b5de4d6895a1937786317ca9, State: Initialized, Role: FOLLOWER
I20260926 09:36:25.981578 19566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811e0bd9b5de4d6895a1937786317ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } }
I20260926 09:36:25.981664 19566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:25.981695 19566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:25.981734 19566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:25.982416 19566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811e0bd9b5de4d6895a1937786317ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } }
I20260926 09:36:25.982519 19566 leader_election.cc:302] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 811e0bd9b5de4d6895a1937786317ca9; no voters:
I20260926 09:36:25.982659 19566 leader_election.cc:288] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:25.982724 19571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:25.982877 19571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [term 1 LEADER]: Becoming Leader. State: Replica: 811e0bd9b5de4d6895a1937786317ca9, State: Running, Role: LEADER
I20260926 09:36:25.982887 19566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:25.982962 19571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811e0bd9b5de4d6895a1937786317ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } }
I20260926 09:36:25.983297 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 811e0bd9b5de4d6895a1937786317ca9. Latest consensus state: current_term: 1 leader_uuid: "811e0bd9b5de4d6895a1937786317ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811e0bd9b5de4d6895a1937786317ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } } }
I20260926 09:36:25.983609 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:25.983347 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "811e0bd9b5de4d6895a1937786317ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811e0bd9b5de4d6895a1937786317ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } } }
I20260926 09:36:25.983732 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:25.983816 19580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:25.984309 19580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:36:25 dist-test-slave-21h8 krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:25.987403 19580 catalog_manager.cc:1347] Generated new cluster ID: ec27ab5710b74fffa430c4fd6da0ef3e
I20260926 09:36:25.987457 19580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:25.990913 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:25.979475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.0.98.62:43603)
0926 09:36:25.979636 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:25.979655 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:25.979691 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:36:25.980441 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:25.980450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:25.980511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:25.980669 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:25.981260 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:25.983007 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:25.984413 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:25.985023 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:25.985133 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:25.987826 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:25.987830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:25.987830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:25.987842 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:25.989437 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:25.989796 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:25.989799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:25.989799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:25.989899 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:25.990104 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:25.990105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:25.990106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:25.990331 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:25.990495 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:25.990573 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 09:36:25.990733 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260926 09:36:26.005957 19580 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:26.006692 19580 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:26.012121 19580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9: Generated new TSK 0
I20260926 09:36:26.012333 19580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:36:26.041796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:26.142244 19592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:26.142431 19592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:26.142452 19592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:26.143610 19592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:26.143656 19592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:26.143682 19592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:36:26.145156 19592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:26.145393 19592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:26.145618 19592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:26.150714 19600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:26.150730 19598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:26.150876 19592 server_base.cc:1034] running on GCE node
W20260926 09:36:26.150765 19597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:26.151266 19592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:26.151546 19592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:26.152702 19592 hybrid_clock.cc:648] HybridClock initialized: now 1790415386152695 us; error 31 us; skew 500 ppm
Sep 26 09:36:26 dist-test-slave-21h8 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:26.155264 19592 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:36:26.155715 19592 webserver.cc:460] Webserver started at http://127.0.98.1:43471/ using document root <none> and password file <none>
I20260926 09:36:26.155949 19592 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:26.156018 19592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:26.156174 19592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:26.157276 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "e021ce474d1c404889fe92b699727bf7"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "15f90b342557fb833fe2866b218d5d09"
server_key_iv: "7bd298d5b506e1a40f8285a5ff879362"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.157626 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e021ce474d1c404889fe92b699727bf7"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "15f90b342557fb833fe2866b218d5d09"
server_key_iv: "7bd298d5b506e1a40f8285a5ff879362"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.158890 19592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:26.159696 19607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.159883 19592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:36:26.159981 19592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "e021ce474d1c404889fe92b699727bf7"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "15f90b342557fb833fe2866b218d5d09"
server_key_iv: "7bd298d5b506e1a40f8285a5ff879362"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.160056 19592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:26.174156 19592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:26.176174 19592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:26.176333 19592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:26.176559 19592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:26.176887 19592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:26.176941 19592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.176976 19592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:26.177003 19592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.182789 19592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45975
I20260926 09:36:26.182839 19720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45975 every 8 connection(s)
I20260926 09:36:26.183154 19592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:36:26.186991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 19592
I20260926 09:36:26.187107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:36:26.187358 392 external_mini_cluster.cc:1428] Setting key 3fd3211e0f7dd1a915c8ac410ba77723
Sep 26 09:36:26 dist-test-slave-21h8 krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:26.193925 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.183960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40097 (local address 127.0.98.62:43603)
0926 09:36:26.184057 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.184078 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.184312 (+ 234us) server_negotiation.cc:407] Connection header received
0926 09:36:26.185288 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.185289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.185299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.185353 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:26.185908 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.186406 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.187774 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.188095 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.188170 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.190850 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:26.190852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:26.190852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:26.190859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:26.192416 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:26.192941 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:26.192942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:26.192942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:26.192999 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:26.193291 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:26.193291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:26.193291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:26.193388 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:26.193440 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.193795 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.193860 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 09:36:26.194195 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.184007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43603 (local address 127.0.98.1:40097)
0926 09:36:26.184187 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.184223 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.185199 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.185407 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.185411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.185656 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:26.185840 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.185843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.186484 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.186485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.187688 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.187690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.188194 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.188195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.188352 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.188834 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:26.188844 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:26.190739 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:26.192504 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:26.192511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:26.192514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:26.192865 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:26.193068 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:26.193069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:26.193070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:26.193220 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:26.193455 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:26.193462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:26.193639 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.193779 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.193955 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260926 09:36:26.194809 19721 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43603
I20260926 09:36:26.194926 19721 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:26.195098 19721 heartbeater.cc:511] Master 127.0.98.62:43603 requested a full tablet report, sending...
I20260926 09:36:26.195616 19530 ts_manager.cc:194] Registered new tserver with Master: e021ce474d1c404889fe92b699727bf7 (127.0.98.1:45975)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:36:26.196277 19530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:36:26.212447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:26.291373 19726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:26.291599 19726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:26.291635 19726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:26.292789 19726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:26.292853 19726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:26.292894 19726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:36:26.294340 19726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:26.294605 19726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:26.294827 19726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:26.299631 19731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:26.299590 19732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:26.299787 19726 server_base.cc:1034] running on GCE node
W20260926 09:36:26.299638 19734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:26.300048 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:26.300355 19726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:26.301512 19726 hybrid_clock.cc:648] HybridClock initialized: now 1790415386301498 us; error 26 us; skew 500 ppm
Sep 26 09:36:26 dist-test-slave-21h8 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:26.303865 19726 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:36:26.304270 19726 webserver.cc:460] Webserver started at http://127.0.98.2:41009/ using document root <none> and password file <none>
I20260926 09:36:26.304490 19726 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:26.304555 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:26.304687 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:26.305655 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb681315f8ee4dd3832ebb401ff87254"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "795be4ecd34ac35799fc4043077a1cbb"
server_key_iv: "19d92a5e9b5e7941f2f6acab2c3904cf"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.305990 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb681315f8ee4dd3832ebb401ff87254"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "795be4ecd34ac35799fc4043077a1cbb"
server_key_iv: "19d92a5e9b5e7941f2f6acab2c3904cf"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.307221 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:36:26.308092 19741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.308276 19726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:26.308378 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "cb681315f8ee4dd3832ebb401ff87254"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "795be4ecd34ac35799fc4043077a1cbb"
server_key_iv: "19d92a5e9b5e7941f2f6acab2c3904cf"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.308468 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:26.314688 19726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:26.316475 19726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:26.316635 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:26.316854 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:26.317152 19726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:26.317206 19726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.317248 19726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:26.317278 19726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.323479 19726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32909
I20260926 09:36:26.323532 19854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32909 every 8 connection(s)
I20260926 09:36:26.323817 19726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:36:26.326612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 19726
I20260926 09:36:26.326753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:36:26.326934 392 external_mini_cluster.cc:1428] Setting key 5371cec6f960e97db3d66a692d503691
Sep 26 09:36:26 dist-test-slave-21h8 krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:26.334443 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.324489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43603 (local address 127.0.98.2:53459)
0926 09:36:26.324643 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.324683 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.325628 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.325893 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.325897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.326158 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:26.326338 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.326342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.327258 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.327259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.328634 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.328637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.329095 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.329097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.329256 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.329818 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:26.329826 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:26.331215 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:26.333155 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:26.333160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:26.333162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:26.333363 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:26.333581 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:26.333581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:26.333581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:26.333678 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:26.333997 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:26.334001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:26.334104 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.334182 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.334293 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260926 09:36:26.334440 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.324544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53459 (local address 127.0.98.62:43603)
0926 09:36:26.324599 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.324601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.324781 (+ 180us) server_negotiation.cc:407] Connection header received
0926 09:36:26.325745 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.325746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.325760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.325830 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:26.326456 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.327119 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.328737 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.328973 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.329085 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.331372 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:26.331374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:26.331374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:26.331383 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:26.333069 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:26.333437 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:26.333438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:26.333439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:26.333508 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:26.333744 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:26.333745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:26.333745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:26.333924 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:26.334013 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.334233 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.334325 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:36:26.335002 19855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43603
I20260926 09:36:26.335109 19855 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:26.335282 19855 heartbeater.cc:511] Master 127.0.98.62:43603 requested a full tablet report, sending...
I20260926 09:36:26.335762 19530 ts_manager.cc:194] Registered new tserver with Master: cb681315f8ee4dd3832ebb401ff87254 (127.0.98.2:32909)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:36:26.336278 19530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:36:26.352023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:26.431522 19860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:26.431726 19860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:26.431764 19860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:26.432909 19860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:26.432974 19860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:26.433019 19860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:36:26.434508 19860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:26.434749 19860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:26.434960 19860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:26.440093 19866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:26.440135 19865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:26.440152 19868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:26.440380 19860 server_base.cc:1034] running on GCE node
I20260926 09:36:26.440598 19860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:26.440793 19860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:26.441956 19860 hybrid_clock.cc:648] HybridClock initialized: now 1790415386441942 us; error 31 us; skew 500 ppm
Sep 26 09:36:26 dist-test-slave-21h8 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:26.444396 19860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:36:26.444846 19860 webserver.cc:460] Webserver started at http://127.0.98.3:38717/ using document root <none> and password file <none>
I20260926 09:36:26.445082 19860 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:26.445127 19860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:26.445271 19860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:26.446255 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "f277dbed65304b3bba3cf1a65b7e489a"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "7c5fac69ad81ab20bdad4f5648c6019f"
server_key_iv: "75cccbcffcf94b2812922f2584b260b6"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.446599 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f277dbed65304b3bba3cf1a65b7e489a"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "7c5fac69ad81ab20bdad4f5648c6019f"
server_key_iv: "75cccbcffcf94b2812922f2584b260b6"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.448002 19860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:36:26.448781 19875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.448951 19860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:26.449075 19860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "f277dbed65304b3bba3cf1a65b7e489a"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "7c5fac69ad81ab20bdad4f5648c6019f"
server_key_iv: "75cccbcffcf94b2812922f2584b260b6"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.449151 19860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:26.463455 19860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:26.465183 19860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:26.465339 19860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:26.465557 19860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:26.465840 19860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:26.465890 19860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.465930 19860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:26.465957 19860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.471815 19860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45159
I20260926 09:36:26.471868 19988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45159 every 8 connection(s)
I20260926 09:36:26.472178 19860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:36:26.476583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 19860
I20260926 09:36:26.476691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:36:26.476900 392 external_mini_cluster.cc:1428] Setting key 5675864387ab810a9787657c62ec2bb5
Sep 26 09:36:26 dist-test-slave-21h8 krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:26.481158 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.472832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49983 (local address 127.0.98.62:43603)
0926 09:36:26.472908 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.472910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.473095 (+ 185us) server_negotiation.cc:407] Connection header received
0926 09:36:26.474017 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.474018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.474028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.474081 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:26.474621 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.475133 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.476129 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.476357 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.476433 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.478770 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:26.478771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:26.478772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:26.478779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:26.480060 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:26.480405 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:26.480406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:26.480406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:26.480455 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:26.480657 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:26.480657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:26.480658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:26.480751 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:26.480801 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.481037 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.481096 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:36:26.481289 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.472802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43603 (local address 127.0.98.3:49983)
0926 09:36:26.472980 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.473015 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.473891 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.474132 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.474135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.474375 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:26.474553 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.474557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.475207 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.475208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.476026 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.476028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.476457 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.476458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.476605 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.477213 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:26.477221 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:26.478655 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:26.480128 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:26.480134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:26.480135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:26.480341 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:26.480512 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:26.480512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:26.480513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:26.480602 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:26.480804 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:26.480808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:26.480911 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.480989 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.481096 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260926 09:36:26.481776 19989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43603
I20260926 09:36:26.481876 19989 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:26.482038 19989 heartbeater.cc:511] Master 127.0.98.62:43603 requested a full tablet report, sending...
I20260926 09:36:26.482414 19530 ts_manager.cc:194] Registered new tserver with Master: f277dbed65304b3bba3cf1a65b7e489a (127.0.98.3:45159)
I20260926 09:36:26.482764 19530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49983
I20260926 09:36:26.488688 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:36:26.495618 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.490065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.0.98.62:43603)
0926 09:36:26.490121 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.490123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.490218 (+ 95us) server_negotiation.cc:407] Connection header received
0926 09:36:26.490276 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.490277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.490284 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.490335 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:26.490883 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.491462 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.492134 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.492301 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.492348 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.493355 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:26.493356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:26.493356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:26.493362 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:26.494692 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:26.494985 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:26.494986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:26.494986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:26.495035 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:26.495174 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:26.495174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:26.495175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:26.495286 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:26.495424 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.495502 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.495561 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:36:26.497087 19530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:36:26.498631 19530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:36:26.504963 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.500998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.0.98.1:45975)
0926 09:36:26.501223 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.501226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.501237 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:36:26.501273 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.501275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.501290 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.501495 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:26.501577 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.502658 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.503913 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.504603 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.504685 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.504746 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.504869 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.504905 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":116,"threads_started":1}
I20260926 09:36:26.504974 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.500723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45975 (local address 127.0.0.1:55084)
0926 09:36:26.501075 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.501083 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.501119 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.501380 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.501381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.501382 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:26.501420 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.501421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.502763 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.502764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.503830 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.503831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.504686 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.504687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.504764 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.504778 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.504849 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.504899 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":44,"threads_started":1}
I20260926 09:36:26.505299 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.500712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32909 (local address 127.0.0.1:55814)
0926 09:36:26.500941 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.500952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.501004 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.501496 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.501497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.501500 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:26.501549 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.501551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.502448 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.502449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.503739 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.503742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.505086 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.505087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.505148 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.505152 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.505201 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.505238 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":55,"threads_started":1}
I20260926 09:36:26.505524 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.501155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.0.98.2:32909)
0926 09:36:26.501325 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.501330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.501340 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:26.501380 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.501382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.501398 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.501626 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:26.501699 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.502344 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.503827 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.504607 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.504685 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.504747 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.505398 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.505445 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260926 09:36:26.505740 19655 tablet_service.cc:1467] Processing CreateTablet for tablet 038479debb0c4b73ab11196f19d085ee (DEFAULT_TABLE table=test-table [id=b900ca2f9a224013b77d084ad3e649a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:26.505877 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.500710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45159 (local address 127.0.0.1:50574)
0926 09:36:26.500804 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.500842 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.500901 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.501333 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.501336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.501338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:26.501403 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.501405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.503918 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.503918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.504621 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.504622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.505680 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.505681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.505742 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.505746 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.505778 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.505815 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":52,"threads_started":1}
I20260926 09:36:26.506119 19655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038479debb0c4b73ab11196f19d085ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:26.506256 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.500956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.0.98.3:45159)
0926 09:36:26.501139 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.501144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.501155 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:36:26.501195 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.501197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.501214 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.501495 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:26.501576 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.502658 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.504726 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.505574 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.506040 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.506103 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.506128 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.506184 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260926 09:36:26.506157 19789 tablet_service.cc:1467] Processing CreateTablet for tablet 038479debb0c4b73ab11196f19d085ee (DEFAULT_TABLE table=test-table [id=b900ca2f9a224013b77d084ad3e649a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:26.506474 19789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038479debb0c4b73ab11196f19d085ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:26.506807 19923 tablet_service.cc:1467] Processing CreateTablet for tablet 038479debb0c4b73ab11196f19d085ee (DEFAULT_TABLE table=test-table [id=b900ca2f9a224013b77d084ad3e649a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:26.507098 19923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038479debb0c4b73ab11196f19d085ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:26.508827 20003 tablet_bootstrap.cc:492] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7: Bootstrap starting.
I20260926 09:36:26.509505 20003 tablet_bootstrap.cc:654] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:26.509743 20004 tablet_bootstrap.cc:492] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Bootstrap starting.
I20260926 09:36:26.510142 20003 log.cc:824] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:26.510208 20005 tablet_bootstrap.cc:492] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: Bootstrap starting.
I20260926 09:36:26.510545 20004 tablet_bootstrap.cc:654] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:26.510820 20005 tablet_bootstrap.cc:654] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:26.511089 20003 tablet_bootstrap.cc:492] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7: No bootstrap required, opened a new log
I20260926 09:36:26.511165 20003 ts_tablet_manager.cc:1397] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:26.511197 20004 log.cc:824] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:26.511545 20005 log.cc:824] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:26.512143 20004 tablet_bootstrap.cc:492] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: No bootstrap required, opened a new log
I20260926 09:36:26.512249 20004 ts_tablet_manager.cc:1397] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:36:26.512380 20005 tablet_bootstrap.cc:492] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: No bootstrap required, opened a new log
I20260926 09:36:26.512446 20005 ts_tablet_manager.cc:1397] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:26.513207 20003 raft_consensus.cc:348] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.513391 20003 raft_consensus.cc:374] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:26.513430 20003 raft_consensus.cc:729] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e021ce474d1c404889fe92b699727bf7, State: Initialized, Role: FOLLOWER
I20260926 09:36:26.513566 20003 consensus_queue.cc:260] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.513836 20003 ts_tablet_manager.cc:1428] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:26.513957 19721 heartbeater.cc:503] Master 127.0.98.62:43603 was elected leader, sending a full tablet report...
I20260926 09:36:26.514130 20005 raft_consensus.cc:348] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.514325 20005 raft_consensus.cc:374] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:26.514230 20004 raft_consensus.cc:348] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.514371 20005 raft_consensus.cc:729] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f277dbed65304b3bba3cf1a65b7e489a, State: Initialized, Role: FOLLOWER
I20260926 09:36:26.514384 20004 raft_consensus.cc:374] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:26.514418 20004 raft_consensus.cc:729] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb681315f8ee4dd3832ebb401ff87254, State: Initialized, Role: FOLLOWER
I20260926 09:36:26.514500 20005 consensus_queue.cc:260] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.514544 20004 consensus_queue.cc:260] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.514760 20005 ts_tablet_manager.cc:1428] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:36:26.514820 20004 ts_tablet_manager.cc:1428] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:36:26.514864 19989 heartbeater.cc:503] Master 127.0.98.62:43603 was elected leader, sending a full tablet report...
I20260926 09:36:26.514829 19855 heartbeater.cc:503] Master 127.0.98.62:43603 was elected leader, sending a full tablet report...
W20260926 09:36:26.574425 19856 tablet.cc:2369] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:36:26.576931 20010 raft_consensus.cc:482] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:26.577035 20010 raft_consensus.cc:504] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.577376 20010 leader_election.cc:288] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb681315f8ee4dd3832ebb401ff87254 (127.0.98.2:32909), e021ce474d1c404889fe92b699727bf7 (127.0.98.1:45975)
I20260926 09:36:26.580377 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.577571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45975 (local address 127.0.98.3:53589)
0926 09:36:26.577628 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.577654 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.577695 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.577900 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.577901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.577903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:26.577984 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.577986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.578700 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.578702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.579534 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.579536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.580089 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.580090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.580176 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.580180 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.580251 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.580283 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":43}
I20260926 09:36:26.580405 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.577685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53589 (local address 127.0.98.1:45975)
0926 09:36:26.577740 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.577742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.577754 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:36:26.577801 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.577802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.577817 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.577883 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:26.578065 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.578615 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.579625 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.580010 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.580068 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.580077 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.580276 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.580326 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:36:26.580459 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.577665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49035 (local address 127.0.98.2:32909)
0926 09:36:26.577719 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.577721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.577880 (+ 159us) server_negotiation.cc:407] Connection header received
0926 09:36:26.577921 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.577922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.577934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.577987 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:26.578202 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.578636 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.579700 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.580071 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.580150 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.580161 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.580352 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.580389 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:36:26.580528 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.577585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32909 (local address 127.0.98.3:49035)
0926 09:36:26.577776 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.577788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.577855 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.578037 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.578039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.578041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:26.578124 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.578126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.578749 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.578750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.579611 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.579614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.580162 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.580163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.580269 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.580273 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.580337 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.580374 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":117,"threads_started":1}
I20260926 09:36:26.580734 19675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038479debb0c4b73ab11196f19d085ee" candidate_uuid: "f277dbed65304b3bba3cf1a65b7e489a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e021ce474d1c404889fe92b699727bf7" is_pre_election: true
I20260926 09:36:26.580890 19675 raft_consensus.cc:2454] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f277dbed65304b3bba3cf1a65b7e489a in term 0.
I20260926 09:36:26.580967 19809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038479debb0c4b73ab11196f19d085ee" candidate_uuid: "f277dbed65304b3bba3cf1a65b7e489a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb681315f8ee4dd3832ebb401ff87254" is_pre_election: true
I20260926 09:36:26.581133 19809 raft_consensus.cc:2454] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f277dbed65304b3bba3cf1a65b7e489a in term 0.
I20260926 09:36:26.581128 19878 leader_election.cc:302] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e021ce474d1c404889fe92b699727bf7, f277dbed65304b3bba3cf1a65b7e489a; no voters:
I20260926 09:36:26.581278 20010 raft_consensus.cc:2781] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:36:26.581349 20010 raft_consensus.cc:482] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:36:26.581382 20010 raft_consensus.cc:3037] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:26.582088 20010 raft_consensus.cc:504] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.582247 20010 leader_election.cc:288] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [CANDIDATE]: Term 1 election: Requested vote from peers cb681315f8ee4dd3832ebb401ff87254 (127.0.98.2:32909), e021ce474d1c404889fe92b699727bf7 (127.0.98.1:45975)
I20260926 09:36:26.582420 19809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038479debb0c4b73ab11196f19d085ee" candidate_uuid: "f277dbed65304b3bba3cf1a65b7e489a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb681315f8ee4dd3832ebb401ff87254"
I20260926 09:36:26.582476 19675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038479debb0c4b73ab11196f19d085ee" candidate_uuid: "f277dbed65304b3bba3cf1a65b7e489a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e021ce474d1c404889fe92b699727bf7"
I20260926 09:36:26.582508 19809 raft_consensus.cc:3037] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:26.582573 19675 raft_consensus.cc:3037] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:26.583343 19809 raft_consensus.cc:2454] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f277dbed65304b3bba3cf1a65b7e489a in term 1.
I20260926 09:36:26.583400 19675 raft_consensus.cc:2454] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f277dbed65304b3bba3cf1a65b7e489a in term 1.
I20260926 09:36:26.583583 19878 leader_election.cc:302] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e021ce474d1c404889fe92b699727bf7, f277dbed65304b3bba3cf1a65b7e489a; no voters:
I20260926 09:36:26.583693 20010 raft_consensus.cc:2781] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:26.583879 20010 raft_consensus.cc:686] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 1 LEADER]: Becoming Leader. State: Replica: f277dbed65304b3bba3cf1a65b7e489a, State: Running, Role: LEADER
I20260926 09:36:26.584004 20010 consensus_queue.cc:237] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } }
I20260926 09:36:26.584909 19528 catalog_manager.cc:5495] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a reported cstate change: term changed from 0 to 1, leader changed from <none> to f277dbed65304b3bba3cf1a65b7e489a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f277dbed65304b3bba3cf1a65b7e489a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f277dbed65304b3bba3cf1a65b7e489a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 } health_report { overall_health: UNKNOWN } } }
I20260926 09:36:26.599159 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.596805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.0.98.3:45159)
0926 09:36:26.596848 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.596851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.596860 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:36:26.596931 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.596931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.596955 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.597033 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:26.597198 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.597635 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.598402 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.598601 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.598683 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.598827 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:26.598916 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:26.599009 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.599028 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.599048 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:36:26.600895 19809 raft_consensus.cc:1261] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 1 FOLLOWER]: Refusing update from remote peer f277dbed65304b3bba3cf1a65b7e489a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:26.600895 19675 raft_consensus.cc:1261] T 038479debb0c4b73ab11196f19d085ee P e021ce474d1c404889fe92b699727bf7 [term 1 FOLLOWER]: Refusing update from remote peer f277dbed65304b3bba3cf1a65b7e489a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:26.601264 20013 consensus_queue.cc:1035] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e021ce474d1c404889fe92b699727bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:26.601477 20010 consensus_queue.cc:1035] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb681315f8ee4dd3832ebb401ff87254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:26.604723 20018 mvcc.cc:205] Tried to move back new op lower bound from 7333541423514701824 to 7333541423448993792. Current Snapshot: MvccSnapshot[applied={T|T < 7333541423514701824}]
I20260926 09:36:26.605933 19528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:26.606029 19528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:26.606767 19528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 811e0bd9b5de4d6895a1937786317ca9: Sending DeleteTablet for 3 replicas of tablet 038479debb0c4b73ab11196f19d085ee
I20260926 09:36:26.607087 19923 tablet_service.cc:1514] Processing DeleteTablet for tablet 038479debb0c4b73ab11196f19d085ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50574
I20260926 09:36:26.607141 19655 tablet_service.cc:1514] Processing DeleteTablet for tablet 038479debb0c4b73ab11196f19d085ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55084
I20260926 09:36:26.607167 19789 tablet_service.cc:1514] Processing DeleteTablet for tablet 038479debb0c4b73ab11196f19d085ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55814
I20260926 09:36:26.607472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 19592
I20260926 09:36:26.607496 20027 tablet_replica.cc:330] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: stopping tablet replica
I20260926 09:36:26.609897 20027 raft_consensus.cc:2229] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:36:26.610031 20027 raft_consensus.cc:2258] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:26.610212 20026 tablet_replica.cc:330] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: stopping tablet replica
I20260926 09:36:26.610301 20026 raft_consensus.cc:2229] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:36:26.610383 20026 raft_consensus.cc:2258] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:26.610512 20027 ts_tablet_manager.cc:1905] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:26.612287 20027 ts_tablet_manager.cc:1918] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:36:26.612358 20027 log.cc:1196] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-1/wal/wals/038479debb0c4b73ab11196f19d085ee
I20260926 09:36:26.612628 20027 ts_tablet_manager.cc:1939] T 038479debb0c4b73ab11196f19d085ee P cb681315f8ee4dd3832ebb401ff87254: Deleting consensus metadata
I20260926 09:36:26.613049 19514 catalog_manager.cc:4841] TS cb681315f8ee4dd3832ebb401ff87254 (127.0.98.2:32909): tablet 038479debb0c4b73ab11196f19d085ee (table test-table [id=b900ca2f9a224013b77d084ad3e649a4]) successfully deleted
W20260926 09:36:26.613962 19516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45975 (error 108)
I20260926 09:36:26.614540 20026 ts_tablet_manager.cc:1905] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:36:26.615897 20026 ts_tablet_manager.cc:1918] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 09:36:26.615912 19997 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:26.614327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45975 (local address 127.0.0.1:55096)
0926 09:36:26.614395 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.614409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.614457 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.615865 (+ 1408us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 09:36:26.616034 19516 catalog_manager.cc:4584] TS e021ce474d1c404889fe92b699727bf7 (127.0.98.1:45975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 038479debb0c4b73ab11196f19d085ee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:36:26.616080 20026 log.cc:1196] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: Deleting WAL directory at /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790415146110679-392-0/minicluster-data/ts-2/wal/wals/038479debb0c4b73ab11196f19d085ee
I20260926 09:36:26.616358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 19726
I20260926 09:36:26.616400 20026 ts_tablet_manager.cc:1939] T 038479debb0c4b73ab11196f19d085ee P f277dbed65304b3bba3cf1a65b7e489a: Deleting consensus metadata
I20260926 09:36:26.616957 19517 catalog_manager.cc:4841] TS f277dbed65304b3bba3cf1a65b7e489a (127.0.98.3:45159): tablet 038479debb0c4b73ab11196f19d085ee (table test-table [id=b900ca2f9a224013b77d084ad3e649a4]) successfully deleted
I20260926 09:36:26.623122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 19860
I20260926 09:36:26.627843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 19498
2026-09-26T09:36:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T09:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:26Z Disabled control of system clock
I20260926 09:36:26.655545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:36:26.735670 20033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:26.735872 20033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:26.735893 20033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:26.736958 20033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:26.736990 20033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:26.737003 20033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:26.737016 20033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:26.737028 20033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:26.738250 20033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:26.738420 20033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:26.738627 20033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:26.743359 20041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:26.743480 20033 server_base.cc:1034] running on GCE node
W20260926 09:36:26.743397 20039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:26.743381 20038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:26.743839 20033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:26.744172 20033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:26.745324 20033 hybrid_clock.cc:648] HybridClock initialized: now 1790415386745300 us; error 38 us; skew 500 ppm
I20260926 09:36:26.746493 20033 webserver.cc:460] Webserver started at http://127.0.98.62:46213/ using document root <none> and password file <none>
I20260926 09:36:26.746675 20033 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:26.746718 20033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:26.746804 20033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:26.747890 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "1dc4c3c5c849c7af3f42e8bb0cb44aa7"
server_key_iv: "ee456276cd5068a9626e54b8f5eccb4e"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.748198 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "1dc4c3c5c849c7af3f42e8bb0cb44aa7"
server_key_iv: "ee456276cd5068a9626e54b8f5eccb4e"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.749400 20033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:36:26.750047 20047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.750226 20033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:26.750296 20033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "1dc4c3c5c849c7af3f42e8bb0cb44aa7"
server_key_iv: "ee456276cd5068a9626e54b8f5eccb4e"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.750345 20033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:26.765535 20033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:26.765789 20033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:26.765905 20033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:26.770059 20033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39185
I20260926 09:36:26.770160 20099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39185 every 8 connection(s)
I20260926 09:36:26.770448 20033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:26.771585 20100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:26.775130 20100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e: Bootstrap starting.
I20260926 09:36:26.775849 20100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:26.776221 20100 log.cc:824] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:26.777099 20100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e: No bootstrap required, opened a new log
I20260926 09:36:26.778750 20100 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260926 09:36:26.778908 20100 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:26.778959 20100 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0aa1d1d3d9a43cebebd285fd5e78e5e, State: Initialized, Role: FOLLOWER
I20260926 09:36:26.779111 20100 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260926 09:36:26.779191 20100 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:26.779225 20100 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:26.779273 20100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:26.779542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 20033
I20260926 09:36:26.779666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:26.779873 392 external_mini_cluster.cc:1428] Setting key 37eee9efe263ed851568c291269e608d
I20260926 09:36:26.779874 20100 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260926 09:36:26.780005 20100 leader_election.cc:302] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0aa1d1d3d9a43cebebd285fd5e78e5e; no voters:
I20260926 09:36:26.780189 20100 leader_election.cc:288] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:26.780252 20103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:26.780465 20100 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:26.780463 20103 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [term 1 LEADER]: Becoming Leader. State: Replica: c0aa1d1d3d9a43cebebd285fd5e78e5e, State: Running, Role: LEADER
I20260926 09:36:26.780610 20103 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260926 09:36:26.781104 20104 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } } }
I20260926 09:36:26.781203 20104 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:26.781338 20105 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0aa1d1d3d9a43cebebd285fd5e78e5e. Latest consensus state: current_term: 1 leader_uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aa1d1d3d9a43cebebd285fd5e78e5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } } }
I20260926 09:36:26.781510 20105 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:26.781719 20114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:26.782303 20114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:26.784158 20114 catalog_manager.cc:1347] Generated new cluster ID: 53c570302aba451e8082a492be063569
I20260926 09:36:26.784222 20114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:26.787763 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.780605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.0.98.62:39185)
0926 09:36:26.781069 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.781078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.781115 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:36:26.781675 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.781684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.781768 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.782277 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:26.782381 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.783882 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.785216 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.785856 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.786319 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.786800 (+ 481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:26.786804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:26.786805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:26.787025 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:26.787065 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:26.787080 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:26.787425 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.787459 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.787540 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":73,"threads_started":1}
I20260926 09:36:26.790885 20114 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:26.791512 20114 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:26.799472 20114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e: Generated new TSK 0
I20260926 09:36:26.799657 20114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:26.806413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--builtin_ntp_servers=127.0.98.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:36:26.886981 20124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:26.887171 20124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:26.887192 20124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:26.888386 20124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:26.888430 20124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:26.888463 20124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:36:26.889789 20124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:26.890030 20124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:26.890331 20124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:26.896224 20129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:26.896238 20130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:26.896234 20132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:26.896677 20124 server_base.cc:1034] running on GCE node
I20260926 09:36:26.896878 20124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:26.897145 20124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:26.898299 20124 hybrid_clock.cc:648] HybridClock initialized: now 1790415386898285 us; error 31 us; skew 500 ppm
I20260926 09:36:26.899662 20124 webserver.cc:460] Webserver started at http://127.0.98.1:38383/ using document root <none> and password file <none>
I20260926 09:36:26.899923 20124 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:26.899977 20124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:26.900126 20124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:26.901213 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/data/instance:
uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "bb36ad7870181bd6dadc90c9e9d7607d"
server_key_iv: "b90bc00c8e384a32af50cea3e2b486dc"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.901572 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "bb36ad7870181bd6dadc90c9e9d7607d"
server_key_iv: "b90bc00c8e384a32af50cea3e2b486dc"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.902870 20124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:26.903666 20138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.903865 20124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:26.903970 20124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/wal
uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e"
format_stamp: "Formatted at 2026-09-26 09:36:26 on dist-test-slave-21h8"
server_key: "bb36ad7870181bd6dadc90c9e9d7607d"
server_key_iv: "b90bc00c8e384a32af50cea3e2b486dc"
server_key_version: "encryptionkey@0"
I20260926 09:36:26.904057 20124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:26.923462 20124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:26.923781 20124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:26.923938 20124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:26.924180 20124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:26.924533 20124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:26.924588 20124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.924623 20124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:26.924650 20124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:26.930104 20124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43133
I20260926 09:36:26.930187 20251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43133 every 8 connection(s)
I20260926 09:36:26.930508 20124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:36:26.931157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 20124
I20260926 09:36:26.931268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:36:26.931530 392 external_mini_cluster.cc:1428] Setting key 911c87525a3231fcf0f6bae3c3fd4a57
I20260926 09:36:26.932682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--builtin_ntp_servers=127.0.98.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:36:26.936268 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.931281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56945 (local address 127.0.98.62:39185)
0926 09:36:26.931363 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:26.931366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:26.931716 (+ 350us) server_negotiation.cc:407] Connection header received
0926 09:36:26.932344 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:26.932345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:26.932359 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:26.932406 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:26.933302 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.933787 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.934791 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.934995 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.935072 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.935970 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:26.935972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:26.935972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:26.935977 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:26.935998 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:26.936002 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:26.936064 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:26.936160 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 09:36:26.936191 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 09:36:26.936491 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:26.931249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39185 (local address 127.0.98.1:56945)
0926 09:36:26.931570 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:26.931630 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:26.932262 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:26.932493 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:26.932497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:26.932508 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:26.933211 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.933219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.933889 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.933890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.934683 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:26.934686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:26.935082 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:26.935083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:26.935213 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:26.935809 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:26.935809 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:26.935836 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:26.935859 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:26.936065 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:26.936071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:26.936075 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:26.936162 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 09:36:26.936264 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":151,"threads_started":1}
I20260926 09:36:26.936955 20252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39185
I20260926 09:36:26.937065 20252 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:26.937282 20252 heartbeater.cc:511] Master 127.0.98.62:39185 requested a full tablet report, sending...
I20260926 09:36:26.937780 20064 ts_manager.cc:194] Registered new tserver with Master: f838bfafc74e4bb29c4cc11cb2df1d8e (127.0.98.1:43133)
I20260926 09:36:26.938428 20064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56945
W20260926 09:36:27.013495 20255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:27.013679 20255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:27.013700 20255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:27.014832 20255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:27.014871 20255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:27.014896 20255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:36:27.016299 20255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:27.016528 20255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:27.016817 20255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:27.021709 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:27.021726 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:27.021986 20255 server_base.cc:1034] running on GCE node
W20260926 09:36:27.022020 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:27.022298 20255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:27.022542 20255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:27.023708 20255 hybrid_clock.cc:648] HybridClock initialized: now 1790415387023683 us; error 39 us; skew 500 ppm
I20260926 09:36:27.024865 20255 webserver.cc:460] Webserver started at http://127.0.98.2:33067/ using document root <none> and password file <none>
I20260926 09:36:27.025121 20255 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:27.025190 20255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:27.025333 20255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:27.026312 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb3fd1059d8e4b48ae729240fb46e26c"
format_stamp: "Formatted at 2026-09-26 09:36:27 on dist-test-slave-21h8"
server_key: "dfd61fe76e6b076fef83e4e74e4baf8e"
server_key_iv: "61b0d6ec7fbacac734bc8219940b46b9"
server_key_version: "encryptionkey@0"
I20260926 09:36:27.026645 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb3fd1059d8e4b48ae729240fb46e26c"
format_stamp: "Formatted at 2026-09-26 09:36:27 on dist-test-slave-21h8"
server_key: "dfd61fe76e6b076fef83e4e74e4baf8e"
server_key_iv: "61b0d6ec7fbacac734bc8219940b46b9"
server_key_version: "encryptionkey@0"
I20260926 09:36:27.028038 20255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:27.028823 20269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:27.029024 20255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:36:27.029119 20255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/wal
uuid: "fb3fd1059d8e4b48ae729240fb46e26c"
format_stamp: "Formatted at 2026-09-26 09:36:27 on dist-test-slave-21h8"
server_key: "dfd61fe76e6b076fef83e4e74e4baf8e"
server_key_iv: "61b0d6ec7fbacac734bc8219940b46b9"
server_key_version: "encryptionkey@0"
I20260926 09:36:27.029217 20255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:27.059010 20255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:27.059355 20255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:27.059514 20255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:27.059751 20255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:27.060079 20255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:27.060134 20255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:27.060175 20255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:27.060204 20255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:27.065713 20255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33249
I20260926 09:36:27.065760 20382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33249 every 8 connection(s)
I20260926 09:36:27.066071 20255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:36:27.067585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 20255
I20260926 09:36:27.067682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:36:27.067878 392 external_mini_cluster.cc:1428] Setting key f5fc35cd44412d45c5a9cecd646185a4
I20260926 09:36:27.068923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--builtin_ntp_servers=127.0.98.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:36:27.072512 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.066785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58535 (local address 127.0.98.62:39185)
0926 09:36:27.066840 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.066843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.067152 (+ 309us) server_negotiation.cc:407] Connection header received
0926 09:36:27.067842 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.067843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.067855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.067973 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:27.068793 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.069372 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.070689 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.071018 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.071099 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.072170 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:27.072171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:27.072172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:27.072176 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:27.072198 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:27.072204 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:27.072268 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.072363 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.072396 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:36:27.072629 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.066798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39185 (local address 127.0.98.2:58535)
0926 09:36:27.067015 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.067049 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:27.067722 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:27.068053 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:27.068057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:27.068066 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:27.068702 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.068708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.069473 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.069474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.070579 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.070581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.071128 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.071130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.071297 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.071969 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:27.071970 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:27.072014 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:27.072040 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:27.072279 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:27.072284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:27.072287 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:27.072346 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:36:27.072438 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260926 09:36:27.073102 20383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39185
I20260926 09:36:27.073197 20383 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:27.073385 20383 heartbeater.cc:511] Master 127.0.98.62:39185 requested a full tablet report, sending...
I20260926 09:36:27.073797 20064 ts_manager.cc:194] Registered new tserver with Master: fb3fd1059d8e4b48ae729240fb46e26c (127.0.98.2:33249)
I20260926 09:36:27.074345 20064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58535
W20260926 09:36:27.150962 20386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:27.151150 20386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:27.151172 20386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:27.152339 20386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:27.152379 20386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:27.152405 20386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:36:27.153777 20386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:27.153982 20386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:27.154240 20386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:27.159209 20392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:27.159221 20391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:27.159276 20386 server_base.cc:1034] running on GCE node
W20260926 09:36:27.159220 20394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:27.159786 20386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:27.160054 20386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:27.161221 20386 hybrid_clock.cc:648] HybridClock initialized: now 1790415387161194 us; error 44 us; skew 500 ppm
I20260926 09:36:27.162488 20386 webserver.cc:460] Webserver started at http://127.0.98.3:46267/ using document root <none> and password file <none>
I20260926 09:36:27.162741 20386 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:27.162788 20386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:27.162961 20386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:27.164067 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e19ac45b0f4419aae7235d339a94f57"
format_stamp: "Formatted at 2026-09-26 09:36:27 on dist-test-slave-21h8"
server_key: "c533791fa24725714ef86f146f40c33b"
server_key_iv: "a833b5c693acdba9045cee1ad5e8fd3f"
server_key_version: "encryptionkey@0"
I20260926 09:36:27.164410 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e19ac45b0f4419aae7235d339a94f57"
format_stamp: "Formatted at 2026-09-26 09:36:27 on dist-test-slave-21h8"
server_key: "c533791fa24725714ef86f146f40c33b"
server_key_iv: "a833b5c693acdba9045cee1ad5e8fd3f"
server_key_version: "encryptionkey@0"
I20260926 09:36:27.165683 20386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:36:27.166453 20400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:27.166640 20386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:27.166735 20386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/wal
uuid: "3e19ac45b0f4419aae7235d339a94f57"
format_stamp: "Formatted at 2026-09-26 09:36:27 on dist-test-slave-21h8"
server_key: "c533791fa24725714ef86f146f40c33b"
server_key_iv: "a833b5c693acdba9045cee1ad5e8fd3f"
server_key_version: "encryptionkey@0"
I20260926 09:36:27.166793 20386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:27.181870 20386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:27.182201 20386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:27.182350 20386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:27.182570 20386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:36:27.182925 20386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:36:27.182977 20386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:27.183012 20386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:36:27.183039 20386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:27.188723 20386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35157
I20260926 09:36:27.188797 20513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35157 every 8 connection(s)
I20260926 09:36:27.189123 20386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:36:27.193894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 20386
I20260926 09:36:27.193996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790415146110679-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:36:27.194207 392 external_mini_cluster.cc:1428] Setting key ef195335886d0f5b64d2453e456ae911
I20260926 09:36:27.194553 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.189885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38485 (local address 127.0.98.62:39185)
0926 09:36:27.189931 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.189934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.190149 (+ 215us) server_negotiation.cc:407] Connection header received
0926 09:36:27.190698 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.190699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.190711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.190766 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:27.191521 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.192034 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.193039 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.193243 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.193322 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.194223 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:27.194225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:27.194225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:27.194230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:27.194247 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:27.194253 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:27.194314 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.194434 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.194464 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:27.194769 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.189852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39185 (local address 127.0.98.3:38485)
0926 09:36:27.190024 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.190058 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:27.190570 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:27.190837 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:27.190841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:27.190850 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:27.191430 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.191436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.192146 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.192147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.192945 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.192947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.193328 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.193329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.193475 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.194088 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:36:27.194089 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:27.194117 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:36:27.194140 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:27.194332 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:27.194337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:27.194340 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:27.194419 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:36:27.194521 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260926 09:36:27.195250 20514 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39185
I20260926 09:36:27.195384 20514 heartbeater.cc:464] Registering TS with master...
I20260926 09:36:27.195562 20514 heartbeater.cc:511] Master 127.0.98.62:39185 requested a full tablet report, sending...
I20260926 09:36:27.195957 20063 ts_manager.cc:194] Registered new tserver with Master: 3e19ac45b0f4419aae7235d339a94f57 (127.0.98.3:35157)
I20260926 09:36:27.196343 20063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38485
I20260926 09:36:27.206094 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:36:27.210511 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.207651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.0.98.62:39185)
0926 09:36:27.207728 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.207730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.207795 (+ 65us) server_negotiation.cc:407] Connection header received
0926 09:36:27.207863 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.207864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.207877 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.207927 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:27.208099 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.208573 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.209338 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.209545 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.209620 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.210255 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:27.210257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:27.210257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:27.210261 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:27.210276 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:27.210280 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:27.210347 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.210400 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.210425 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:36:27.212112 20063 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:36:27.213644 20063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:36:27.220248 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.216075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35157 (local address 127.0.0.1:56438)
0926 09:36:27.216287 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.216298 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:27.216357 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:27.216736 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:27.216737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:27.216740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:27.216785 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.216787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.217872 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.217872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.219206 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.219208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.219940 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.219941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.220013 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.220026 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 09:36:27.220103 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:36:27.220154 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":139,"thread_start_us":69,"threads_started":1}
I20260926 09:36:27.220352 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.215779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33249 (local address 127.0.0.1:59752)
0926 09:36:27.215943 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.215983 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:27.216048 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:27.216263 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:27.216265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:27.216268 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:27.216330 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.216332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.217779 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.217780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.219099 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.219101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.220109 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.220110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.220168 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.220170 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:36:27.220213 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 09:36:27.220243 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":52,"thread_start_us":63,"threads_started":1}
I20260926 09:36:27.220400 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.215869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.0.98.2:33249)
0926 09:36:27.216060 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.216065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.216085 (+ 20us) server_negotiation.cc:407] Connection header received
0926 09:36:27.216124 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.216126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.216141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.216408 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:27.216465 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.217676 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.219190 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.220030 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.220115 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.220219 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.220247 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.220288 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260926 09:36:27.220643 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.216284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.0.98.3:35157)
0926 09:36:27.216490 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.216494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.216505 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:36:27.216547 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.216549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.216565 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.216886 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:27.216963 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.217761 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.219350 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.219844 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.219940 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.220038 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.220498 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.220550 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260926 09:36:27.220903 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.215858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43133 (local address 127.0.0.1:49818)
0926 09:36:27.216186 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.216195 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:27.216258 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:27.216510 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:27.216511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:27.216513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:27.216544 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.216545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.217826 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.217827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.219206 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.219208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.220541 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.220542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.220605 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.220608 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:27.220763 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 09:36:27.220812 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":53,"thread_start_us":110,"threads_started":1}
I20260926 09:36:27.221189 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.215996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.0.98.1:43133)
0926 09:36:27.216204 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.216209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.216316 (+ 107us) server_negotiation.cc:407] Connection header received
0926 09:36:27.216361 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.216364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.216379 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.216443 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:27.216614 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.217763 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.219356 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.220133 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.220200 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.220282 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.220707 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.221102 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260926 09:36:27.221210 20317 tablet_service.cc:1467] Processing CreateTablet for tablet d55edca6260143efa20fc9adb51fad29 (DEFAULT_TABLE table=test-table [id=eb57c890adb74df9afa5fb1cc192b7f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:27.221596 20317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55edca6260143efa20fc9adb51fad29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:27.221534 20448 tablet_service.cc:1467] Processing CreateTablet for tablet d55edca6260143efa20fc9adb51fad29 (DEFAULT_TABLE table=test-table [id=eb57c890adb74df9afa5fb1cc192b7f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:27.221889 20448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55edca6260143efa20fc9adb51fad29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:27.221815 20186 tablet_service.cc:1467] Processing CreateTablet for tablet d55edca6260143efa20fc9adb51fad29 (DEFAULT_TABLE table=test-table [id=eb57c890adb74df9afa5fb1cc192b7f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:36:27.222137 20186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55edca6260143efa20fc9adb51fad29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:27.224609 20528 tablet_bootstrap.cc:492] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: Bootstrap starting.
I20260926 09:36:27.225257 20528 tablet_bootstrap.cc:654] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:27.225610 20529 tablet_bootstrap.cc:492] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57: Bootstrap starting.
I20260926 09:36:27.225610 20528 log.cc:824] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:27.225991 20530 tablet_bootstrap.cc:492] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e: Bootstrap starting.
I20260926 09:36:27.226294 20529 tablet_bootstrap.cc:654] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:27.226317 20528 tablet_bootstrap.cc:492] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: No bootstrap required, opened a new log
I20260926 09:36:27.226394 20528 ts_tablet_manager.cc:1397] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:36:27.226666 20530 tablet_bootstrap.cc:654] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:27.226647 20529 log.cc:824] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:27.227003 20530 log.cc:824] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:27.227509 20529 tablet_bootstrap.cc:492] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57: No bootstrap required, opened a new log
I20260926 09:36:27.227634 20529 ts_tablet_manager.cc:1397] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:27.228215 20530 tablet_bootstrap.cc:492] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e: No bootstrap required, opened a new log
I20260926 09:36:27.228287 20530 ts_tablet_manager.cc:1397] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:36:27.228955 20528 raft_consensus.cc:348] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.229106 20528 raft_consensus.cc:374] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:27.229149 20528 raft_consensus.cc:729] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3fd1059d8e4b48ae729240fb46e26c, State: Initialized, Role: FOLLOWER
I20260926 09:36:27.229281 20528 consensus_queue.cc:260] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.229552 20528 ts_tablet_manager.cc:1428] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:27.229683 20383 heartbeater.cc:503] Master 127.0.98.62:39185 was elected leader, sending a full tablet report...
I20260926 09:36:27.230471 20529 raft_consensus.cc:348] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.230659 20529 raft_consensus.cc:374] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:27.230708 20529 raft_consensus.cc:729] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e19ac45b0f4419aae7235d339a94f57, State: Initialized, Role: FOLLOWER
I20260926 09:36:27.230827 20529 consensus_queue.cc:260] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.231088 20529 ts_tablet_manager.cc:1428] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:27.231029 20530 raft_consensus.cc:348] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.231119 20514 heartbeater.cc:503] Master 127.0.98.62:39185 was elected leader, sending a full tablet report...
I20260926 09:36:27.231180 20530 raft_consensus.cc:374] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:27.231225 20530 raft_consensus.cc:729] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f838bfafc74e4bb29c4cc11cb2df1d8e, State: Initialized, Role: FOLLOWER
I20260926 09:36:27.231357 20530 consensus_queue.cc:260] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.231619 20530 ts_tablet_manager.cc:1428] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:36:27.231710 20252 heartbeater.cc:503] Master 127.0.98.62:39185 was elected leader, sending a full tablet report...
W20260926 09:36:27.316679 20384 tablet.cc:2369] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:36:27.342490 20535 raft_consensus.cc:482] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:36:27.342813 20535 raft_consensus.cc:504] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.343369 20535 leader_election.cc:288] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb3fd1059d8e4b48ae729240fb46e26c (127.0.98.2:33249), f838bfafc74e4bb29c4cc11cb2df1d8e (127.0.98.1:43133)
I20260926 09:36:27.348032 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.343553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33249 (local address 127.0.98.3:53921)
0926 09:36:27.343607 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.343619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:27.343667 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:27.343821 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:27.343822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:27.343824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:27.343895 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.343897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.345226 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.345227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.346788 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.346791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.347687 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.347688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.347810 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.347814 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:36:27.347907 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 09:36:27.347959 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:36:27.348068 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.343626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53921 (local address 127.0.98.2:33249)
0926 09:36:27.343697 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.343699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.343707 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:36:27.343734 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.343735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.343746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.343799 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:27.343972 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.344808 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.346874 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.347610 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.347689 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.347700 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.347915 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.347967 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:36:27.348407 20337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55edca6260143efa20fc9adb51fad29" candidate_uuid: "3e19ac45b0f4419aae7235d339a94f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" is_pre_election: true
I20260926 09:36:27.348563 20337 raft_consensus.cc:2454] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e19ac45b0f4419aae7235d339a94f57 in term 0.
I20260926 09:36:27.348798 20404 leader_election.cc:302] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e19ac45b0f4419aae7235d339a94f57, fb3fd1059d8e4b48ae729240fb46e26c; no voters:
I20260926 09:36:27.348909 20535 raft_consensus.cc:2781] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:36:27.348943 20535 raft_consensus.cc:482] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:36:27.348964 20535 raft_consensus.cc:3037] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:27.349766 20535 raft_consensus.cc:504] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.349864 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.345076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43133 (local address 127.0.98.3:54339)
0926 09:36:27.345235 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.345254 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:27.345301 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:27.346183 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:27.346184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:27.346187 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:36:27.346263 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.346265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.347878 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.347879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.349004 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:27.349006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.349592 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.349593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:27.349679 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.349682 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:36:27.349740 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:36:27.349782 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":16,"thread_start_us":94,"threads_started":1}
I20260926 09:36:27.349915 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.345725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54339 (local address 127.0.98.1:43133)
0926 09:36:27.345806 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.345808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.345819 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:36:27.345861 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.345862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.345875 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.346151 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:36:27.346898 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.347812 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.349123 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.349505 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.349563 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.349569 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.349767 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.349818 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:36:27.350063 20337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55edca6260143efa20fc9adb51fad29" candidate_uuid: "3e19ac45b0f4419aae7235d339a94f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3fd1059d8e4b48ae729240fb46e26c"
I20260926 09:36:27.350121 20337 raft_consensus.cc:3037] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:27.350605 20205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55edca6260143efa20fc9adb51fad29" candidate_uuid: "3e19ac45b0f4419aae7235d339a94f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e"
I20260926 09:36:27.350605 20206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55edca6260143efa20fc9adb51fad29" candidate_uuid: "3e19ac45b0f4419aae7235d339a94f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" is_pre_election: true
I20260926 09:36:27.350760 20205 raft_consensus.cc:3037] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:27.351011 20337 raft_consensus.cc:2454] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e19ac45b0f4419aae7235d339a94f57 in term 1.
I20260926 09:36:27.349963 20535 leader_election.cc:288] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [CANDIDATE]: Term 1 election: Requested vote from peers fb3fd1059d8e4b48ae729240fb46e26c (127.0.98.2:33249), f838bfafc74e4bb29c4cc11cb2df1d8e (127.0.98.1:43133)
I20260926 09:36:27.351159 20404 leader_election.cc:302] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e19ac45b0f4419aae7235d339a94f57, fb3fd1059d8e4b48ae729240fb46e26c; no voters:
I20260926 09:36:27.351336 20535 raft_consensus.cc:2781] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:27.351397 20535 raft_consensus.cc:686] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 1 LEADER]: Becoming Leader. State: Replica: 3e19ac45b0f4419aae7235d339a94f57, State: Running, Role: LEADER
I20260926 09:36:27.351476 20535 consensus_queue.cc:237] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260926 09:36:27.351940 20205 raft_consensus.cc:2454] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e19ac45b0f4419aae7235d339a94f57 in term 1.
I20260926 09:36:27.352355 20063 catalog_manager.cc:5495] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e19ac45b0f4419aae7235d339a94f57 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3e19ac45b0f4419aae7235d339a94f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e19ac45b0f4419aae7235d339a94f57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } health_report { overall_health: HEALTHY } } }
I20260926 09:36:27.368638 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.366221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.0.98.3:35157)
0926 09:36:27.366274 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:27.366276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:27.366307 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:36:27.366398 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:27.366398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:27.366415 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:27.366481 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:27.366562 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.367044 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.367959 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:27.368148 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:27.368210 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:27.368336 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:27.368458 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:27.368521 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:27.368553 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 09:36:27.368568 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:36:27.370517 20205 raft_consensus.cc:1261] T d55edca6260143efa20fc9adb51fad29 P f838bfafc74e4bb29c4cc11cb2df1d8e [term 1 FOLLOWER]: Refusing update from remote peer 3e19ac45b0f4419aae7235d339a94f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:27.370560 20337 raft_consensus.cc:1261] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 1 FOLLOWER]: Refusing update from remote peer 3e19ac45b0f4419aae7235d339a94f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:27.370842 20535 consensus_queue.cc:1035] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f838bfafc74e4bb29c4cc11cb2df1d8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:27.371012 20538 consensus_queue.cc:1035] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3fd1059d8e4b48ae729240fb46e26c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:27.373273 20541 mvcc.cc:205] Tried to move back new op lower bound from 7333541426666827776 to 7333541426592198656. Current Snapshot: MvccSnapshot[applied={T|T < 7333541426666827776}]
I20260926 09:36:27.374310 20543 mvcc.cc:205] Tried to move back new op lower bound from 7333541426666827776 to 7333541426592198656. Current Snapshot: MvccSnapshot[applied={T|T < 7333541426666827776}]
I20260926 09:36:27.374478 20542 mvcc.cc:205] Tried to move back new op lower bound from 7333541426666827776 to 7333541426592198656. Current Snapshot: MvccSnapshot[applied={T|T < 7333541426666827776 or (T in {7333541426666827776})}]
I20260926 09:36:27.375536 20063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:27.375623 20063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:36:27.376304 20063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0aa1d1d3d9a43cebebd285fd5e78e5e: Sending DeleteTablet for 3 replicas of tablet d55edca6260143efa20fc9adb51fad29
I20260926 09:36:27.376659 20186 tablet_service.cc:1514] Processing DeleteTablet for tablet d55edca6260143efa20fc9adb51fad29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:27 UTC) from {username='slave'} at 127.0.0.1:49818
I20260926 09:36:27.376652 20317 tablet_service.cc:1514] Processing DeleteTablet for tablet d55edca6260143efa20fc9adb51fad29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:27 UTC) from {username='slave'} at 127.0.0.1:59752
I20260926 09:36:27.376681 20448 tablet_service.cc:1514] Processing DeleteTablet for tablet d55edca6260143efa20fc9adb51fad29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:36:27 UTC) from {username='slave'} at 127.0.0.1:56438
I20260926 09:36:27.376966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 20124
I20260926 09:36:27.379776 20551 tablet_replica.cc:330] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: stopping tablet replica
I20260926 09:36:27.379858 20551 raft_consensus.cc:2229] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:36:27.381810 20551 raft_consensus.cc:2258] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:27.382308 20551 ts_tablet_manager.cc:1905] T d55edca6260143efa20fc9adb51fad29 P fb3fd1059d8e4b48ae729240fb46e26c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 09:36:27.383702 20050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43133 (error 108)
I20260926 09:36:27.383965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 20255
I20260926 09:36:27.384667 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.384047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43133 (local address 127.0.0.1:49830)
0926 09:36:27.384572 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.384615 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504}
W20260926 09:36:27.385918 20050 catalog_manager.cc:4584] TS f838bfafc74e4bb29c4cc11cb2df1d8e (127.0.98.1:43133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d55edca6260143efa20fc9adb51fad29: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43133: connect: Connection refused (error 111)
I20260926 09:36:27.389109 20552 tablet_replica.cc:330] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57: stopping tablet replica
I20260926 09:36:27.389261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 20386
I20260926 09:36:27.389269 20552 raft_consensus.cc:2229] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:36:27.389293 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.389199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33249 (local address 127.0.0.1:59760)
0926 09:36:27.389241 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.389263 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 09:36:27.389453 20552 raft_consensus.cc:2258] T d55edca6260143efa20fc9adb51fad29 P 3e19ac45b0f4419aae7235d339a94f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:36:27.393780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 20033
I20260926 09:36:27.393796 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:27.393688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35157 (local address 127.0.0.1:56450)
0926 09:36:27.393739 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:27.393773 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
2026-09-26T09:36:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (764 ms)
[----------] 6 tests from EncryptionPolicyTest (8285 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:27 dist-test-slave-21h8 krb5kdc[20556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:27 dist-test-slave-21h8 krb5kdc[20556](info): set up 2 sockets
Sep 26 09:36:27 dist-test-slave-21h8 krb5kdc[20556](info): commencing operation
krb5kdc: starting...
W20260926 09:36:28.427289 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:28 dist-test-slave-21h8 krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:28.545799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:28.624751 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:28.624989 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:28.625028 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:28.626178 20569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:28.626248 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:28.626274 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:28.626307 20569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:28.626331 20569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:28.627990 20569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:28.628258 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:28.628477 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:28.633172 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:28.633167 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:28.633164 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:28.633872 20569 server_base.cc:1034] running on GCE node
I20260926 09:36:28.634116 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:28.634352 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:28.635522 20569 hybrid_clock.cc:648] HybridClock initialized: now 1790415388635507 us; error 44 us; skew 500 ppm
Sep 26 09:36:28 dist-test-slave-21h8 krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:28.637836 20569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:28.638249 20569 webserver.cc:460] Webserver started at http://127.0.0.1:44289/ using document root <none> and password file <none>
I20260926 09:36:28.638487 20569 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:28.638554 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:28.638702 20569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:28.639819 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "2031d4cf3c114a7785d7a09ea226e7ca"
format_stamp: "Formatted at 2026-09-26 09:36:28 on dist-test-slave-21h8"
server_key: "d63caf7003bb2e42760eba7637bcb335"
server_key_iv: "2afc3136e12a9afd6708ad9307b667c5"
server_key_version: "encryptionkey@0"
I20260926 09:36:28.640166 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "2031d4cf3c114a7785d7a09ea226e7ca"
format_stamp: "Formatted at 2026-09-26 09:36:28 on dist-test-slave-21h8"
server_key: "d63caf7003bb2e42760eba7637bcb335"
server_key_iv: "2afc3136e12a9afd6708ad9307b667c5"
server_key_version: "encryptionkey@0"
I20260926 09:36:28.641475 20569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:36:28.642204 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:28.642369 20569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:28.642484 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "2031d4cf3c114a7785d7a09ea226e7ca"
format_stamp: "Formatted at 2026-09-26 09:36:28 on dist-test-slave-21h8"
server_key: "d63caf7003bb2e42760eba7637bcb335"
server_key_iv: "2afc3136e12a9afd6708ad9307b667c5"
server_key_version: "encryptionkey@0"
I20260926 09:36:28.642560 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:28.655045 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:28.656927 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:28.657087 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:28.660936 20569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32825
I20260926 09:36:28.660985 20636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32825 every 8 connection(s)
I20260926 09:36:28.661336 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:28.662225 20637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:28.664554 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca: Bootstrap starting.
I20260926 09:36:28.665115 20637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:28.665400 20637 log.cc:824] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:28.666065 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca: No bootstrap required, opened a new log
I20260926 09:36:28.667689 20637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031d4cf3c114a7785d7a09ea226e7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32825 } }
I20260926 09:36:28.667824 20637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:28.667846 20637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2031d4cf3c114a7785d7a09ea226e7ca, State: Initialized, Role: FOLLOWER
I20260926 09:36:28.667995 20637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031d4cf3c114a7785d7a09ea226e7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32825 } }
I20260926 09:36:28.668063 20637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:28.668085 20637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:28.668136 20637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:28.668635 20637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031d4cf3c114a7785d7a09ea226e7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32825 } }
I20260926 09:36:28.668720 20637 leader_election.cc:302] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2031d4cf3c114a7785d7a09ea226e7ca; no voters:
I20260926 09:36:28.668892 20637 leader_election.cc:288] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:28.669021 20640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:28.669193 20640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [term 1 LEADER]: Becoming Leader. State: Replica: 2031d4cf3c114a7785d7a09ea226e7ca, State: Running, Role: LEADER
I20260926 09:36:28.669242 20637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:28.669291 20640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031d4cf3c114a7785d7a09ea226e7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32825 } }
I20260926 09:36:28.669672 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2031d4cf3c114a7785d7a09ea226e7ca. Latest consensus state: current_term: 1 leader_uuid: "2031d4cf3c114a7785d7a09ea226e7ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031d4cf3c114a7785d7a09ea226e7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32825 } } }
I20260926 09:36:28.669759 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:28.669901 20641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2031d4cf3c114a7785d7a09ea226e7ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031d4cf3c114a7785d7a09ea226e7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32825 } } }
I20260926 09:36:28.670014 20641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:28.670246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 20569
I20260926 09:36:28.670375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:28.670562 392 external_mini_cluster.cc:1428] Setting key fc16855a299104685c24905c1d96991f
W20260926 09:36:28.671924 20658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:36:28.671990 20658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:36:28.672040 20648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:28.672222 20648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:28.673611 20648 catalog_manager.cc:1347] Generated new cluster ID: 6df663a979694153bfe2683077803136
I20260926 09:36:28.673666 20648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:36:28 dist-test-slave-21h8 krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:36:28.681382 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:28.671079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42187 (local address 127.0.0.1:32825)
0926 09:36:28.671292 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:28.671303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:28.671362 (+ 59us) server_negotiation.cc:407] Connection header received
0926 09:36:28.671482 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:28.671490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:28.671549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:28.671713 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:28.672463 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:28.673582 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:28.674625 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:28.675016 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:28.675093 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:28.677566 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:28.677570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:28.677571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:28.677583 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:28.679587 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:28.680004 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:28.680007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:28.680007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:28.680137 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:28.680361 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:28.680363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:28.680363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:28.680667 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:28.680829 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:28.680941 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 09:36:28.681170 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20260926 09:36:28.682649 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:28.689303 20648 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:28.689810 20648 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:28.691054 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:28.684028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53615 (local address 127.0.0.1:32825)
0926 09:36:28.684088 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:28.684091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:28.684246 (+ 155us) server_negotiation.cc:407] Connection header received
0926 09:36:28.684277 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:28.684278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:28.684287 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:28.684356 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:28.684883 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:28.685645 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:28.686587 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:28.686904 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:28.686975 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:28.688299 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:28.688301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:28.688301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:28.688308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:28.689914 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:28.690279 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:28.690280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:28.690281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:28.690341 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:28.690553 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:28.690554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:28.690554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:28.690693 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:28.690807 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:28.690907 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 09:36:28.690969 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:36:28.697477 20648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2031d4cf3c114a7785d7a09ea226e7ca: Generated new TSK 0
I20260926 09:36:28.697633 20648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:28.702306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 20569
W20260926 09:36:28.706300 20554 connection.cc:535] client connection to 127.0.0.1:32825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32825): Connection reset by peer (error 104)
2026-09-26T09:36:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:28 dist-test-slave-21h8 krb5kdc[20668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:28 dist-test-slave-21h8 krb5kdc[20668](info): set up 2 sockets
Sep 26 09:36:28 dist-test-slave-21h8 krb5kdc[20668](info): commencing operation
krb5kdc: starting...
W20260926 09:36:29.735550 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:29 dist-test-slave-21h8 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:29.852022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:29.931205 20681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:29.931465 20681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:29.931505 20681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:29.932636 20681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:29.932705 20681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:29.932730 20681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:29.932757 20681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:29.932783 20681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:29.934195 20681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:29.934454 20681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:29.934691 20681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:29.939419 20687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:29.939534 20681 server_base.cc:1034] running on GCE node
W20260926 09:36:29.939424 20686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:29.939419 20689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:29.939898 20681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:29.940152 20681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:29.941294 20681 hybrid_clock.cc:648] HybridClock initialized: now 1790415389941285 us; error 29 us; skew 500 ppm
Sep 26 09:36:29 dist-test-slave-21h8 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:29.952328 20681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:29.952813 20681 webserver.cc:460] Webserver started at http://127.0.0.1:43321/ using document root <none> and password file <none>
I20260926 09:36:29.953044 20681 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:29.953143 20681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:29.953294 20681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:29.954351 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "1384eb02755f41c3aeec6783eedcc9f5"
format_stamp: "Formatted at 2026-09-26 09:36:29 on dist-test-slave-21h8"
server_key: "68061805772063d93f1fbd8d1c4a035a"
server_key_iv: "ff59011e60c7c08ea1d88814026e8d56"
server_key_version: "encryptionkey@0"
I20260926 09:36:29.954716 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "1384eb02755f41c3aeec6783eedcc9f5"
format_stamp: "Formatted at 2026-09-26 09:36:29 on dist-test-slave-21h8"
server_key: "68061805772063d93f1fbd8d1c4a035a"
server_key_iv: "ff59011e60c7c08ea1d88814026e8d56"
server_key_version: "encryptionkey@0"
I20260926 09:36:29.956197 20681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:29.956945 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:29.957116 20681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:29.957217 20681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "1384eb02755f41c3aeec6783eedcc9f5"
format_stamp: "Formatted at 2026-09-26 09:36:29 on dist-test-slave-21h8"
server_key: "68061805772063d93f1fbd8d1c4a035a"
server_key_iv: "ff59011e60c7c08ea1d88814026e8d56"
server_key_version: "encryptionkey@0"
I20260926 09:36:29.957296 20681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:29.978803 20681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:29.980806 20681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:29.980968 20681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:29.984938 20681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35315
I20260926 09:36:29.984982 20748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35315 every 8 connection(s)
I20260926 09:36:29.985345 20681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:29.986179 20749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:29.986563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 20681
I20260926 09:36:29.986673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:29.986881 392 external_mini_cluster.cc:1428] Setting key 422c322f5d0a49f3153597a736602970
I20260926 09:36:29.989068 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5: Bootstrap starting.
I20260926 09:36:29.989729 20749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:29.990033 20749 log.cc:824] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:29.990729 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5: No bootstrap required, opened a new log
I20260926 09:36:29.992478 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:29.987414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57925 (local address 127.0.0.1:35315)
0926 09:36:29.987619 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:29.987627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:29.987651 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:36:29.988063 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:29.988069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:29.988123 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:29.988234 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:29.988503 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:29.989427 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:29.990533 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:29.990982 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:29.991063 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:29.991773 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:29.991778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:29.991778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:29.992001 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:29.992043 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:29.992055 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:29.992128 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:29.992232 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 09:36:29.992275 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260926 09:36:29.993060 20749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1384eb02755f41c3aeec6783eedcc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35315 } }
I20260926 09:36:29.993263 20749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:29.993310 20749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1384eb02755f41c3aeec6783eedcc9f5, State: Initialized, Role: FOLLOWER
I20260926 09:36:29.993434 20749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1384eb02755f41c3aeec6783eedcc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35315 } }
I20260926 09:36:29.993528 20749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:29.993547 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:29.993570 20749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:29.993621 20749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:29.994129 20749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1384eb02755f41c3aeec6783eedcc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35315 } }
I20260926 09:36:29.994210 20749 leader_election.cc:302] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1384eb02755f41c3aeec6783eedcc9f5; no voters:
I20260926 09:36:29.994396 20749 leader_election.cc:288] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:29.994432 20754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:29.994547 20754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [term 1 LEADER]: Becoming Leader. State: Replica: 1384eb02755f41c3aeec6783eedcc9f5, State: Running, Role: LEADER
I20260926 09:36:29.994642 20749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:29.994686 20754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1384eb02755f41c3aeec6783eedcc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35315 } }
I20260926 09:36:29.995018 20756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1384eb02755f41c3aeec6783eedcc9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1384eb02755f41c3aeec6783eedcc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35315 } } }
I20260926 09:36:29.995121 20756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:29.995237 20758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1384eb02755f41c3aeec6783eedcc9f5. Latest consensus state: current_term: 1 leader_uuid: "1384eb02755f41c3aeec6783eedcc9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1384eb02755f41c3aeec6783eedcc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35315 } } }
I20260926 09:36:29.995296 20758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:29.995471 20765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:29.995962 20765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:29.998633 20765 catalog_manager.cc:1347] Generated new cluster ID: 16fd126fa9384fc9873bda464d7d4ba1
I20260926 09:36:29.998679 20765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:29.999589 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:29.995129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55189 (local address 127.0.0.1:35315)
0926 09:36:29.995164 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:29.995165 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:29.996275 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 09:36:29.996376 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:29.996377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:29.996387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:29.996442 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:29.996564 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:29.997234 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:29.998227 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:29.998410 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:29.998508 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:29.999180 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:29.999181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:29.999181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:29.999185 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:29.999199 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:29.999221 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:29.999298 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:29.999517 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 09:36:29.999536 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:36:30.015808 20765 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:30.016433 20765 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:30.026690 20765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1384eb02755f41c3aeec6783eedcc9f5: Generated new TSK 0
I20260926 09:36:30.026891 20765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:30.036021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 20681
W20260926 09:36:30.040406 20666 connection.cc:535] client connection to 127.0.0.1:35315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35315): Connection reset by peer (error 104)
2026-09-26T09:36:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:30 dist-test-slave-21h8 krb5kdc[20780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:30 dist-test-slave-21h8 krb5kdc[20780](info): set up 2 sockets
Sep 26 09:36:30 dist-test-slave-21h8 krb5kdc[20780](info): commencing operation
krb5kdc: starting...
W20260926 09:36:31.067588 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:31 dist-test-slave-21h8 krb5kdc[20780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:31.180156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:31.259724 20793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:31.259915 20793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:31.259938 20793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:31.261060 20793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:31.261103 20793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:31.261117 20793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:31.261155 20793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:31.261183 20793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:31.262591 20793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:31.262861 20793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:31.263079 20793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:31.267720 20799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:31.267774 20793 server_base.cc:1034] running on GCE node
W20260926 09:36:31.267732 20798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:31.267732 20801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:31.268190 20793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:31.268481 20793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:31.269657 20793 hybrid_clock.cc:648] HybridClock initialized: now 1790415391269647 us; error 31 us; skew 500 ppm
Sep 26 09:36:31 dist-test-slave-21h8 krb5kdc[20780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:31.272014 20793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:31.272370 20793 webserver.cc:460] Webserver started at http://127.0.0.1:45205/ using document root <none> and password file <none>
I20260926 09:36:31.272545 20793 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:31.272583 20793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:31.272666 20793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:31.273573 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "375c39af66e648928f93fc9aef41d4d9"
format_stamp: "Formatted at 2026-09-26 09:36:31 on dist-test-slave-21h8"
server_key: "091c0108d7e83ad261c43c82b2f9ae01"
server_key_iv: "cfb7702ef014bc3ec2564bc0ad950e95"
server_key_version: "encryptionkey@0"
I20260926 09:36:31.273856 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "375c39af66e648928f93fc9aef41d4d9"
format_stamp: "Formatted at 2026-09-26 09:36:31 on dist-test-slave-21h8"
server_key: "091c0108d7e83ad261c43c82b2f9ae01"
server_key_iv: "cfb7702ef014bc3ec2564bc0ad950e95"
server_key_version: "encryptionkey@0"
I20260926 09:36:31.275044 20793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:36:31.275731 20808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:31.275884 20793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:31.275954 20793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "375c39af66e648928f93fc9aef41d4d9"
format_stamp: "Formatted at 2026-09-26 09:36:31 on dist-test-slave-21h8"
server_key: "091c0108d7e83ad261c43c82b2f9ae01"
server_key_iv: "cfb7702ef014bc3ec2564bc0ad950e95"
server_key_version: "encryptionkey@0"
I20260926 09:36:31.276003 20793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:31.278936 20793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:31.280614 20793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:31.280719 20793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:31.284403 20793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42075
I20260926 09:36:31.284449 20860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42075 every 8 connection(s)
I20260926 09:36:31.284721 20793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:31.285605 20861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:31.287725 20861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9: Bootstrap starting.
I20260926 09:36:31.288249 20861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:31.288525 20861 log.cc:824] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:31.289170 20861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9: No bootstrap required, opened a new log
I20260926 09:36:31.290637 20861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c39af66e648928f93fc9aef41d4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } }
I20260926 09:36:31.290760 20861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:31.290782 20861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375c39af66e648928f93fc9aef41d4d9, State: Initialized, Role: FOLLOWER
I20260926 09:36:31.290910 20861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c39af66e648928f93fc9aef41d4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } }
I20260926 09:36:31.290985 20861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:31.291006 20861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:31.291039 20861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:31.291553 20861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c39af66e648928f93fc9aef41d4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } }
I20260926 09:36:31.291636 20861 leader_election.cc:302] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375c39af66e648928f93fc9aef41d4d9; no voters:
I20260926 09:36:31.291755 20861 leader_election.cc:288] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:31.291842 20864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:31.292021 20861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:31.292145 20864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [term 1 LEADER]: Becoming Leader. State: Replica: 375c39af66e648928f93fc9aef41d4d9, State: Running, Role: LEADER
I20260926 09:36:31.292229 20864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c39af66e648928f93fc9aef41d4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } }
I20260926 09:36:31.292593 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375c39af66e648928f93fc9aef41d4d9. Latest consensus state: current_term: 1 leader_uuid: "375c39af66e648928f93fc9aef41d4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c39af66e648928f93fc9aef41d4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } } }
I20260926 09:36:31.292706 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:31.292665 20865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375c39af66e648928f93fc9aef41d4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c39af66e648928f93fc9aef41d4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } } }
I20260926 09:36:31.292769 20865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:31.292935 20873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:31.293434 20873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:31.294425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 20793
I20260926 09:36:31.294543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:31.294716 392 external_mini_cluster.cc:1428] Setting key 23362b22fdc210f84bee16a898d3842b
I20260926 09:36:31.294872 20873 catalog_manager.cc:1347] Generated new cluster ID: 8c878d0b203b444297750e830a57a03a
I20260926 09:36:31.295077 20873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:31.296867 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:31.295380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56567 (local address 127.0.0.1:42075)
0926 09:36:31.295537 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:31.295545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:31.295584 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:36:31.295690 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:31.295696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:31.295722 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:31.295833 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:31.296480 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:31.296483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:31.296484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:31.296487 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:31.296520 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:31.296532 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:31.296586 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:31.296673 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 09:36:31.296708 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260926 09:36:31.297777 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:31.300369 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:31.299275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38415 (local address 127.0.0.1:42075)
0926 09:36:31.299360 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:31.299362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:31.299499 (+ 137us) server_negotiation.cc:407] Connection header received
0926 09:36:31.299556 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:31.299557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:31.299562 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:31.299612 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:31.300175 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:31.300176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:31.300176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:31.300181 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:31.300193 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:31.300197 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:31.300246 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:31.300300 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 09:36:31.300312 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 09:36:31.309428 20873 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:31.309574 20873 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:31.319487 20873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 375c39af66e648928f93fc9aef41d4d9: Generated new TSK 0
I20260926 09:36:31.319703 20873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:31.328358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 20793
2026-09-26T09:36:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:31 dist-test-slave-21h8 krb5kdc[20892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:31 dist-test-slave-21h8 krb5kdc[20892](info): set up 2 sockets
Sep 26 09:36:31 dist-test-slave-21h8 krb5kdc[20892](info): commencing operation
krb5kdc: starting...
W20260926 09:36:32.359249 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:32 dist-test-slave-21h8 krb5kdc[20892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:32.469395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:32.550338 20905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:32.550539 20905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:32.550563 20905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:32.551666 20905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:32.551707 20905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:32.551723 20905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:32.551738 20905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:32.551751 20905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:32.553203 20905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:32.553391 20905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:32.553550 20905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:32.558377 20910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:32.558414 20911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:32.558413 20905 server_base.cc:1034] running on GCE node
W20260926 09:36:32.558384 20913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:32.558815 20905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:32.559094 20905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:32.560259 20905 hybrid_clock.cc:648] HybridClock initialized: now 1790415392560248 us; error 42 us; skew 500 ppm
Sep 26 09:36:32 dist-test-slave-21h8 krb5kdc[20892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:32.562680 20905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:32.563138 20905 webserver.cc:460] Webserver started at http://127.0.0.1:41827/ using document root <none> and password file <none>
I20260926 09:36:32.563421 20905 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:32.563489 20905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:32.563659 20905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:32.564651 20905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "7aea93ba94c84bd3813074db921aba0e"
format_stamp: "Formatted at 2026-09-26 09:36:32 on dist-test-slave-21h8"
server_key: "970bd3ff189fcb4ffaf59dd24d8437e3"
server_key_iv: "6e6caddcb044d42e73074f8620310b70"
server_key_version: "encryptionkey@0"
I20260926 09:36:32.565002 20905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "7aea93ba94c84bd3813074db921aba0e"
format_stamp: "Formatted at 2026-09-26 09:36:32 on dist-test-slave-21h8"
server_key: "970bd3ff189fcb4ffaf59dd24d8437e3"
server_key_iv: "6e6caddcb044d42e73074f8620310b70"
server_key_version: "encryptionkey@0"
I20260926 09:36:32.566250 20905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:32.566931 20920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:32.567114 20905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:32.567209 20905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "7aea93ba94c84bd3813074db921aba0e"
format_stamp: "Formatted at 2026-09-26 09:36:32 on dist-test-slave-21h8"
server_key: "970bd3ff189fcb4ffaf59dd24d8437e3"
server_key_iv: "6e6caddcb044d42e73074f8620310b70"
server_key_version: "encryptionkey@0"
I20260926 09:36:32.567292 20905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:32.595572 20905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:32.597949 20905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:32.598109 20905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:32.601966 20905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34589
I20260926 09:36:32.602003 20972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34589 every 8 connection(s)
I20260926 09:36:32.602347 20905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:32.603170 20973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:32.603791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 20905
I20260926 09:36:32.603917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:32.604096 392 external_mini_cluster.cc:1428] Setting key bd21f9d532b5e165d0dfb7f867ae1dc9
I20260926 09:36:32.606173 20973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e: Bootstrap starting.
I20260926 09:36:32.606822 20973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:32.607149 20973 log.cc:824] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:32.607832 20973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e: No bootstrap required, opened a new log
I20260926 09:36:32.609917 20973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aea93ba94c84bd3813074db921aba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } }
I20260926 09:36:32.610050 20973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:32.610072 20973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aea93ba94c84bd3813074db921aba0e, State: Initialized, Role: FOLLOWER
I20260926 09:36:32.610208 20973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aea93ba94c84bd3813074db921aba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } }
I20260926 09:36:32.610272 20973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:32.610293 20973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:32.610368 20973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 0 FOLLOWER]: Advancing to term 1
Sep 26 09:36:32 dist-test-slave-21h8 krb5kdc[20892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:36:32.611053 20973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aea93ba94c84bd3813074db921aba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } }
I20260926 09:36:32.611152 20973 leader_election.cc:302] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aea93ba94c84bd3813074db921aba0e; no voters:
I20260926 09:36:32.611338 20973 leader_election.cc:288] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:32.611379 20978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:32.611524 20978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [term 1 LEADER]: Becoming Leader. State: Replica: 7aea93ba94c84bd3813074db921aba0e, State: Running, Role: LEADER
I20260926 09:36:32.611594 20973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:32.611647 20978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aea93ba94c84bd3813074db921aba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } }
I20260926 09:36:32.612028 20978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aea93ba94c84bd3813074db921aba0e. Latest consensus state: current_term: 1 leader_uuid: "7aea93ba94c84bd3813074db921aba0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aea93ba94c84bd3813074db921aba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } } }
I20260926 09:36:32.612202 20978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:32.612041 20980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aea93ba94c84bd3813074db921aba0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aea93ba94c84bd3813074db921aba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } } }
I20260926 09:36:32.612463 20980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:32.612656 20986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:32.613191 20986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:32.615144 20986 catalog_manager.cc:1347] Generated new cluster ID: 5c0655325262450f98fa62b7ad22858f
I20260926 09:36:32.615196 20986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:32.615657 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:32.604606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48385 (local address 127.0.0.1:34589)
0926 09:36:32.604805 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:32.604814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:32.604847 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:36:32.605390 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:32.605398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:32.605470 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:32.605615 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:32.606185 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:32.607513 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:32.608323 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:32.608950 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:32.609057 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:32.611715 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:32.611719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:32.611719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:32.611731 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:32.613783 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:32.614155 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:32.614158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:32.614159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:32.614259 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:32.614508 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:32.614508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:32.614509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:32.614835 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:32.614991 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:32.615341 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 09:36:32.615479 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260926 09:36:32.628218 20986 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:32.628326 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:32.628738 20986 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:32.633708 20986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aea93ba94c84bd3813074db921aba0e: Generated new TSK 0
I20260926 09:36:32.633879 20986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:32.636467 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:32.629801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45727 (local address 127.0.0.1:34589)
0926 09:36:32.629842 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:32.629844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:32.629955 (+ 111us) server_negotiation.cc:407] Connection header received
0926 09:36:32.629978 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:32.629978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:32.629988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:32.630040 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:32.630668 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:32.631394 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:32.632479 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:32.632787 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:32.632849 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:32.634194 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:32.634195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:32.634195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:32.634200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:32.635469 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:32.635839 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:32.635840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:32.635840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:32.635886 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:32.636072 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:32.636072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:32.636072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:32.636158 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:32.636228 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:32.636359 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 09:36:32.636411 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:36:32.638199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 20905
2026-09-26T09:36:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:32 dist-test-slave-21h8 krb5kdc[21004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:32 dist-test-slave-21h8 krb5kdc[21004](info): set up 2 sockets
Sep 26 09:36:32 dist-test-slave-21h8 krb5kdc[21004](info): commencing operation
krb5kdc: starting...
W20260926 09:36:33.683200 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:33 dist-test-slave-21h8 krb5kdc[21004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:33.797611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:33.877233 21017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:33.877429 21017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:33.877451 21017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:33.878515 21017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:33.878549 21017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:33.878563 21017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:33.878575 21017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:33.878587 21017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:33.880044 21017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:33.880232 21017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:33.880385 21017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:33.884934 21022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:33.884919 21023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:33.884919 21025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:33.885108 21017 server_base.cc:1034] running on GCE node
I20260926 09:36:33.885363 21017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:33.885690 21017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:33.886832 21017 hybrid_clock.cc:648] HybridClock initialized: now 1790415393886801 us; error 47 us; skew 500 ppm
Sep 26 09:36:33 dist-test-slave-21h8 krb5kdc[21004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:33.889245 21017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:33.889612 21017 webserver.cc:460] Webserver started at http://127.0.0.1:44765/ using document root <none> and password file <none>
I20260926 09:36:33.889797 21017 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:33.889835 21017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:33.889919 21017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:33.890901 21017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "135a20f184c04cc0a6695f10dedbd134"
format_stamp: "Formatted at 2026-09-26 09:36:33 on dist-test-slave-21h8"
server_key: "acb22b8ddc9eccb9ce0895ef6617462b"
server_key_iv: "9ab253a5ca6b4f4c4e44b43b9e7da3b2"
server_key_version: "encryptionkey@0"
I20260926 09:36:33.891201 21017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "135a20f184c04cc0a6695f10dedbd134"
format_stamp: "Formatted at 2026-09-26 09:36:33 on dist-test-slave-21h8"
server_key: "acb22b8ddc9eccb9ce0895ef6617462b"
server_key_iv: "9ab253a5ca6b4f4c4e44b43b9e7da3b2"
server_key_version: "encryptionkey@0"
I20260926 09:36:33.892499 21017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:36:33.893168 21032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:33.893330 21017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:33.893435 21017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "135a20f184c04cc0a6695f10dedbd134"
format_stamp: "Formatted at 2026-09-26 09:36:33 on dist-test-slave-21h8"
server_key: "acb22b8ddc9eccb9ce0895ef6617462b"
server_key_iv: "9ab253a5ca6b4f4c4e44b43b9e7da3b2"
server_key_version: "encryptionkey@0"
I20260926 09:36:33.893508 21017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:33.906260 21017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:33.908106 21017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:33.908263 21017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:33.912322 21017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42067
I20260926 09:36:33.912374 21084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42067 every 8 connection(s)
I20260926 09:36:33.912716 21017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:33.913576 21085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:33.915906 21085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134: Bootstrap starting.
I20260926 09:36:33.916445 21085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:33.916682 21085 log.cc:824] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:33.917335 21085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134: No bootstrap required, opened a new log
I20260926 09:36:33.918908 21085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a20f184c04cc0a6695f10dedbd134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260926 09:36:33.919045 21085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:33.919070 21085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135a20f184c04cc0a6695f10dedbd134, State: Initialized, Role: FOLLOWER
I20260926 09:36:33.919214 21085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a20f184c04cc0a6695f10dedbd134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260926 09:36:33.919299 21085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:33.919395 21085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:33.919438 21085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:33.919937 21085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a20f184c04cc0a6695f10dedbd134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260926 09:36:33.920027 21085 leader_election.cc:302] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135a20f184c04cc0a6695f10dedbd134; no voters:
I20260926 09:36:33.920149 21085 leader_election.cc:288] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:33.920215 21088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:33.920390 21088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [term 1 LEADER]: Becoming Leader. State: Replica: 135a20f184c04cc0a6695f10dedbd134, State: Running, Role: LEADER
I20260926 09:36:33.920418 21085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:33.920476 21088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a20f184c04cc0a6695f10dedbd134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260926 09:36:33.920845 21090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135a20f184c04cc0a6695f10dedbd134. Latest consensus state: current_term: 1 leader_uuid: "135a20f184c04cc0a6695f10dedbd134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a20f184c04cc0a6695f10dedbd134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } } }
I20260926 09:36:33.920833 21089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135a20f184c04cc0a6695f10dedbd134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a20f184c04cc0a6695f10dedbd134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } } }
I20260926 09:36:33.920930 21090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:33.920936 21089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:33.921134 21092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:33.921622 21092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:33.922026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21017
I20260926 09:36:33.922153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:33.922403 392 external_mini_cluster.cc:1428] Setting key 869801a7f6b4e693e422bfc54c3d6c01
I20260926 09:36:33.923590 21092 catalog_manager.cc:1347] Generated new cluster ID: 02469ff95c3e42449f6e320aa4cb9cb0
I20260926 09:36:33.923650 21092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:33.928529 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:33.923050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37707 (local address 127.0.0.1:42067)
0926 09:36:33.923239 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:33.923248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:33.923284 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:36:33.923422 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:33.923429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:33.923484 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:33.923893 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:33.923986 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:33.925285 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:33.926424 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:33.927044 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:33.927164 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:33.927819 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:33.927823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:33.927823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:33.928040 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:33.928080 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:33.928097 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:33.928175 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:33.928238 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 09:36:33.928298 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260926 09:36:33.929550 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:33.934476 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:33.930930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42651 (local address 127.0.0.1:42067)
0926 09:36:33.930972 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:33.930974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:33.931048 (+ 74us) server_negotiation.cc:407] Connection header received
0926 09:36:33.931151 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:33.931152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:33.931162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:33.931196 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:33.931435 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:33.932078 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:33.933082 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:33.933386 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:33.933431 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:33.934186 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:33.934187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:33.934188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:33.934193 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:33.934207 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:33.934211 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:33.934242 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:33.934377 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 09:36:33.934400 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:36:33.941059 21092 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:33.941562 21092 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:33.945381 21092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135a20f184c04cc0a6695f10dedbd134: Generated new TSK 0
I20260926 09:36:33.945585 21092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:33.948056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21017
2026-09-26T09:36:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:33 dist-test-slave-21h8 krb5kdc[21116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:33 dist-test-slave-21h8 krb5kdc[21116](info): set up 2 sockets
Sep 26 09:36:33 dist-test-slave-21h8 krb5kdc[21116](info): commencing operation
krb5kdc: starting...
W20260926 09:36:34.975363 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:35 dist-test-slave-21h8 krb5kdc[21116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:35.093433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:35.171495 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:35.171689 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:35.171712 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:35.172796 21129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:35.172832 21129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:35.172853 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:35.172866 21129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:35.172878 21129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:35.174234 21129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:35.174412 21129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:35.174569 21129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:35.179157 21134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:35.179159 21137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:35.179181 21135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:35.179523 21129 server_base.cc:1034] running on GCE node
I20260926 09:36:35.179796 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:35.180128 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:35.181294 21129 hybrid_clock.cc:648] HybridClock initialized: now 1790415395181271 us; error 37 us; skew 500 ppm
Sep 26 09:36:35 dist-test-slave-21h8 krb5kdc[21116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:35.188167 21129 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:35.188673 21129 webserver.cc:460] Webserver started at http://127.0.0.1:42727/ using document root <none> and password file <none>
I20260926 09:36:35.188933 21129 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:35.189007 21129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:35.189164 21129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:35.190310 21129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "f472369c1b3f4721baee88653311384d"
format_stamp: "Formatted at 2026-09-26 09:36:35 on dist-test-slave-21h8"
server_key: "14dc362db1b8bf84ee1f282ba6abd8b0"
server_key_iv: "1cc6063945b58ce16ebe639211503e61"
server_key_version: "encryptionkey@0"
I20260926 09:36:35.190717 21129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "f472369c1b3f4721baee88653311384d"
format_stamp: "Formatted at 2026-09-26 09:36:35 on dist-test-slave-21h8"
server_key: "14dc362db1b8bf84ee1f282ba6abd8b0"
server_key_iv: "1cc6063945b58ce16ebe639211503e61"
server_key_version: "encryptionkey@0"
I20260926 09:36:35.192277 21129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:35.193028 21144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:35.193207 21129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:35.193320 21129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "f472369c1b3f4721baee88653311384d"
format_stamp: "Formatted at 2026-09-26 09:36:35 on dist-test-slave-21h8"
server_key: "14dc362db1b8bf84ee1f282ba6abd8b0"
server_key_iv: "1cc6063945b58ce16ebe639211503e61"
server_key_version: "encryptionkey@0"
I20260926 09:36:35.193401 21129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:35.197085 21129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:35.199118 21129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:35.199281 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:35.203925 21129 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44663
I20260926 09:36:35.203974 21196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44663 every 8 connection(s)
I20260926 09:36:35.204342 21129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:35.205358 21197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:35.207643 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d: Bootstrap starting.
I20260926 09:36:35.207654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21129
I20260926 09:36:35.207762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:35.207991 392 external_mini_cluster.cc:1428] Setting key 3ef61c079b9295aec43502018c81f29a
I20260926 09:36:35.208283 21197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:35.208647 21197 log.cc:824] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:35.209334 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d: No bootstrap required, opened a new log
I20260926 09:36:35.210546 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:35.208530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56593 (local address 127.0.0.1:44663)
0926 09:36:35.208758 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:35.208769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:35.208803 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:36:35.209259 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:35.209265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:35.209291 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:35.209462 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:35.210103 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:35.210107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:35.210107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:35.210111 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:35.210164 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:35.210175 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:35.210284 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:35.210320 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 09:36:35.210381 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260926 09:36:35.211513 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:35.211792 21197 raft_consensus.cc:348] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472369c1b3f4721baee88653311384d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } }
I20260926 09:36:35.211942 21197 raft_consensus.cc:374] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:35.211987 21197 raft_consensus.cc:729] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f472369c1b3f4721baee88653311384d, State: Initialized, Role: FOLLOWER
I20260926 09:36:35.212100 21197 consensus_queue.cc:260] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472369c1b3f4721baee88653311384d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } }
I20260926 09:36:35.212204 21197 raft_consensus.cc:388] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:35.212248 21197 raft_consensus.cc:482] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:35.212294 21197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:35.212860 21197 raft_consensus.cc:504] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472369c1b3f4721baee88653311384d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } }
I20260926 09:36:35.212975 21197 leader_election.cc:302] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f472369c1b3f4721baee88653311384d; no voters:
I20260926 09:36:35.213148 21197 leader_election.cc:288] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:35.213316 21208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:35.213388 21197 sys_catalog.cc:564] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:35.213588 21208 raft_consensus.cc:686] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [term 1 LEADER]: Becoming Leader. State: Replica: f472369c1b3f4721baee88653311384d, State: Running, Role: LEADER
I20260926 09:36:35.213721 21208 consensus_queue.cc:237] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472369c1b3f4721baee88653311384d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } }
I20260926 09:36:35.214139 21207 sys_catalog.cc:455] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f472369c1b3f4721baee88653311384d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472369c1b3f4721baee88653311384d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } } }
I20260926 09:36:35.214223 21207 sys_catalog.cc:458] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:35.214118 21208 sys_catalog.cc:455] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f472369c1b3f4721baee88653311384d. Latest consensus state: current_term: 1 leader_uuid: "f472369c1b3f4721baee88653311384d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472369c1b3f4721baee88653311384d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } } }
I20260926 09:36:35.214555 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:35.213130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54633 (local address 127.0.0.1:44663)
0926 09:36:35.213172 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:35.213174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:35.213258 (+ 84us) server_negotiation.cc:407] Connection header received
0926 09:36:35.213349 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:35.213350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:35.213354 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:35.213413 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:35.214074 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:35.214075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:35.214075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:35.214080 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:35.214093 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:35.214098 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:35.214131 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:35.214439 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 09:36:35.214453 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:36:35.214488 21216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:35.214576 21208 sys_catalog.cc:458] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:35.215224 21216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:35.216627 21216 catalog_manager.cc:1347] Generated new cluster ID: ea09bc2c9157405a8d77f92898d5f041
I20260926 09:36:35.216677 21216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:35.244817 21216 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:35.245003 21216 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:35.250115 21216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f472369c1b3f4721baee88653311384d: Generated new TSK 0
I20260926 09:36:35.250306 21216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:35.253156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21129
2026-09-26T09:36:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:35 dist-test-slave-21h8 krb5kdc[21228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:35 dist-test-slave-21h8 krb5kdc[21228](info): set up 2 sockets
Sep 26 09:36:35 dist-test-slave-21h8 krb5kdc[21228](info): commencing operation
krb5kdc: starting...
W20260926 09:36:36.279480 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:36 dist-test-slave-21h8 krb5kdc[21228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:36.394377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:36.473045 21241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:36.473274 21241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:36.473311 21241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:36.474438 21241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:36.474502 21241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:36.474529 21241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:36.474561 21241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:36.474584 21241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:36.476578 21241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:36.476826 21241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:36.477064 21241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:36.482046 21246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:36.482117 21241 server_base.cc:1034] running on GCE node
W20260926 09:36:36.482074 21249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:36.482087 21247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:36.482466 21241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:36.482782 21241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:36.483948 21241 hybrid_clock.cc:648] HybridClock initialized: now 1790415396483928 us; error 41 us; skew 500 ppm
Sep 26 09:36:36 dist-test-slave-21h8 krb5kdc[21228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:36.486361 21241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:36.486778 21241 webserver.cc:460] Webserver started at http://127.0.0.1:41291/ using document root <none> and password file <none>
I20260926 09:36:36.486999 21241 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:36.487074 21241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:36.487244 21241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:36.488360 21241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "25b81663ea76420f8c2614b0a45d9786"
format_stamp: "Formatted at 2026-09-26 09:36:36 on dist-test-slave-21h8"
server_key: "e65696c32a27a2d0e11ad0e3a0e9c10c"
server_key_iv: "1274432c806ec62f871a4eda86203689"
server_key_version: "encryptionkey@0"
I20260926 09:36:36.488727 21241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "25b81663ea76420f8c2614b0a45d9786"
format_stamp: "Formatted at 2026-09-26 09:36:36 on dist-test-slave-21h8"
server_key: "e65696c32a27a2d0e11ad0e3a0e9c10c"
server_key_iv: "1274432c806ec62f871a4eda86203689"
server_key_version: "encryptionkey@0"
I20260926 09:36:36.490113 21241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:36:36.490862 21256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:36.491029 21241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:36.491137 21241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "25b81663ea76420f8c2614b0a45d9786"
format_stamp: "Formatted at 2026-09-26 09:36:36 on dist-test-slave-21h8"
server_key: "e65696c32a27a2d0e11ad0e3a0e9c10c"
server_key_iv: "1274432c806ec62f871a4eda86203689"
server_key_version: "encryptionkey@0"
I20260926 09:36:36.491225 21241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:36.508301 21241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:36.510186 21241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:36.510367 21241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:36.514345 21241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32997
I20260926 09:36:36.514393 21308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32997 every 8 connection(s)
I20260926 09:36:36.514751 21241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:36.515631 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:36.517763 21309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786: Bootstrap starting.
I20260926 09:36:36.518283 21309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:36.518579 21309 log.cc:824] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:36.518642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21241
I20260926 09:36:36.518745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:36.518942 392 external_mini_cluster.cc:1428] Setting key cc7cbce9000d88facb30fac98ac3eb26
I20260926 09:36:36.519424 21309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786: No bootstrap required, opened a new log
I20260926 09:36:36.521548 21309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b81663ea76420f8c2614b0a45d9786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260926 09:36:36.521726 21309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:36.521760 21309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b81663ea76420f8c2614b0a45d9786, State: Initialized, Role: FOLLOWER
I20260926 09:36:36.521924 21309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b81663ea76420f8c2614b0a45d9786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260926 09:36:36.522017 21309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:36.522047 21309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:36.522084 21309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:36.522775 21309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b81663ea76420f8c2614b0a45d9786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260926 09:36:36.522879 21309 leader_election.cc:302] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25b81663ea76420f8c2614b0a45d9786; no voters:
I20260926 09:36:36.523018 21309 leader_election.cc:288] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:36.523098 21314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:36.523288 21314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [term 1 LEADER]: Becoming Leader. State: Replica: 25b81663ea76420f8c2614b0a45d9786, State: Running, Role: LEADER
I20260926 09:36:36.523267 21309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:36.523444 21314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b81663ea76420f8c2614b0a45d9786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260926 09:36:36.523728 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25b81663ea76420f8c2614b0a45d9786. Latest consensus state: current_term: 1 leader_uuid: "25b81663ea76420f8c2614b0a45d9786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b81663ea76420f8c2614b0a45d9786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260926 09:36:36.523816 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:36.523739 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25b81663ea76420f8c2614b0a45d9786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b81663ea76420f8c2614b0a45d9786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260926 09:36:36.523964 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786 [sys.catalog]: This master's current role is: LEADER
W20260926 09:36:36.525482 21330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:36:36.525584 21330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:36:36.525671 21321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:36.525841 21321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:36:36 dist-test-slave-21h8 krb5kdc[21228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:36:36.527276 21321 catalog_manager.cc:1347] Generated new cluster ID: 554a3ce28efb42c3a911d5646797b78e
I20260926 09:36:36.527351 21321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:36.530987 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:36.519535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:45801 (local address 127.0.0.1:32997)
0926 09:36:36.519737 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:36.519746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:36.519783 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:36:36.520752 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:36.520761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:36.520826 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:36.520976 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:36.521566 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:36.523259 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:36.524374 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:36.525065 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:36.525515 (+ 450us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:36.527740 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:36.527744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:36.527744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:36.527762 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:36.529421 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:36.529817 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:36.529820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:36.529820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:36.529904 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:36.530159 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:36.530160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:36.530161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:36.530385 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:36.530599 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:36.530693 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 09:36:36.530797 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260926 09:36:36.532001 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:36.534307 21321 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:36.535030 21321 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:36.541030 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:36.533330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:49999 (local address 127.0.0.1:32997)
0926 09:36:36.533366 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:36.533368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:36.534079 (+ 711us) server_negotiation.cc:407] Connection header received
0926 09:36:36.534175 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:36.534177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:36.534188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:36.534244 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:36.534916 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:36.535662 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:36.536617 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:36.536941 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:36.537009 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:36.538246 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:36.538247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:36.538248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:36.538254 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:36.539826 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:36.540212 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:36.540213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:36.540214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:36.540282 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:36.540499 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:36.540499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:36.540500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:36.540616 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:36.540712 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:36.540874 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 09:36:36.540942 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:36:36.543665 21321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25b81663ea76420f8c2614b0a45d9786: Generated new TSK 0
I20260926 09:36:36.543861 21321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:36.546285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21241
W20260926 09:36:36.550491 21226 connection.cc:535] client connection to 127.0.0.1:32997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32997): Connection reset by peer (error 104)
2026-09-26T09:36:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:36 dist-test-slave-21h8 krb5kdc[21340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:36 dist-test-slave-21h8 krb5kdc[21340](info): set up 2 sockets
Sep 26 09:36:36 dist-test-slave-21h8 krb5kdc[21340](info): commencing operation
krb5kdc: starting...
W20260926 09:36:37.583449 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:37 dist-test-slave-21h8 krb5kdc[21340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:37.699069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:37.779199 21353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:37.779486 21353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:37.779531 21353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:37.780694 21353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:37.780762 21353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:37.780786 21353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:37.780807 21353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:37.780833 21353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:37.782289 21353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:37.782538 21353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:37.782753 21353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:37.787616 21358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:37.787657 21359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:37.787616 21361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:37.787798 21353 server_base.cc:1034] running on GCE node
I20260926 09:36:37.788130 21353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:37.788416 21353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:37.789587 21353 hybrid_clock.cc:648] HybridClock initialized: now 1790415397789564 us; error 36 us; skew 500 ppm
Sep 26 09:36:37 dist-test-slave-21h8 krb5kdc[21340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:37.792155 21353 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:37.792649 21353 webserver.cc:460] Webserver started at http://127.0.0.1:35827/ using document root <none> and password file <none>
I20260926 09:36:37.792882 21353 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:37.792929 21353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:37.793087 21353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:37.794121 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6"
format_stamp: "Formatted at 2026-09-26 09:36:37 on dist-test-slave-21h8"
server_key: "a7d90a9e6fe2421ed7db13d98f406b9a"
server_key_iv: "839412500c29cc21fd1d3cff3e04940a"
server_key_version: "encryptionkey@0"
I20260926 09:36:37.794447 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6"
format_stamp: "Formatted at 2026-09-26 09:36:37 on dist-test-slave-21h8"
server_key: "a7d90a9e6fe2421ed7db13d98f406b9a"
server_key_iv: "839412500c29cc21fd1d3cff3e04940a"
server_key_version: "encryptionkey@0"
I20260926 09:36:37.795892 21353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:37.796633 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:37.796825 21353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:36:37.796902 21353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6"
format_stamp: "Formatted at 2026-09-26 09:36:37 on dist-test-slave-21h8"
server_key: "a7d90a9e6fe2421ed7db13d98f406b9a"
server_key_iv: "839412500c29cc21fd1d3cff3e04940a"
server_key_version: "encryptionkey@0"
I20260926 09:36:37.797003 21353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:37.806427 21353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:37.808291 21353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:37.808452 21353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:37.812507 21353 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32953
I20260926 09:36:37.812556 21420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32953 every 8 connection(s)
I20260926 09:36:37.812889 21353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:37.813428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21353
I20260926 09:36:37.813544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:37.813748 392 external_mini_cluster.cc:1428] Setting key 8df320b445c86834fdf139f3a56a41b0
I20260926 09:36:37.813927 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:37.816725 21421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6: Bootstrap starting.
I20260926 09:36:37.817512 21421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:37.817801 21421 log.cc:824] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:37.818559 21421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6: No bootstrap required, opened a new log
I20260926 09:36:37.819658 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:37.814241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:39919 (local address 127.0.0.1:32953)
0926 09:36:37.814437 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:37.814449 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:37.814473 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:36:37.814976 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:37.814982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:37.815027 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:37.815171 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:37.815449 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:37.816383 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:37.817388 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:37.817995 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:37.818127 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:37.818953 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:37.818957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:37.818957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:37.819179 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:37.819215 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:37.819232 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:37.819360 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:37.819440 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:36:37.819472 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260926 09:36:37.821166 21421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } }
I20260926 09:36:37.821305 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:37.821370 21421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:37.821424 21421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecdcf4f5d984eaa832e76f77e5ca5a6, State: Initialized, Role: FOLLOWER
I20260926 09:36:37.821547 21421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } }
I20260926 09:36:37.821625 21421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:37.821668 21421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:37.821715 21421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:37.822288 21421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } }
I20260926 09:36:37.822396 21421 leader_election.cc:302] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ecdcf4f5d984eaa832e76f77e5ca5a6; no voters:
I20260926 09:36:37.822568 21421 leader_election.cc:288] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:37.822597 21430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:37.822718 21430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecdcf4f5d984eaa832e76f77e5ca5a6, State: Running, Role: LEADER
I20260926 09:36:37.822816 21421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:37.822849 21430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } }
I20260926 09:36:37.823158 21430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ecdcf4f5d984eaa832e76f77e5ca5a6. Latest consensus state: current_term: 1 leader_uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } } }
I20260926 09:36:37.823189 21431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecdcf4f5d984eaa832e76f77e5ca5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } } }
I20260926 09:36:37.823240 21430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:37.823294 21431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:37.824524 21437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:37.825129 21437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:37.826417 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:37.822765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:46597 (local address 127.0.0.1:32953)
0926 09:36:37.822794 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:37.822795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:37.823162 (+ 367us) server_negotiation.cc:407] Connection header received
0926 09:36:37.823420 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:37.823421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:37.823432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:37.823475 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:37.823632 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:37.824210 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:37.825078 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:37.825365 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:37.825450 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:37.826139 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:37.826141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:37.826141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:37.826145 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:37.826161 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:37.826166 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:37.826238 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:37.826326 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 09:36:37.826338 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":14}
I20260926 09:36:37.827009 21437 catalog_manager.cc:1347] Generated new cluster ID: b7699d9748604e1eb8969b012244476b
I20260926 09:36:37.827059 21437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:37.834004 21437 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:37.834519 21437 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:37.842690 21437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ecdcf4f5d984eaa832e76f77e5ca5a6: Generated new TSK 0
I20260926 09:36:37.842880 21437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:37.845145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21353
W20260926 09:36:37.849328 21338 connection.cc:535] client connection to 127.0.0.1:32953 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32953): Connection reset by peer (error 104)
2026-09-26T09:36:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:37 dist-test-slave-21h8 krb5kdc[21452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:37 dist-test-slave-21h8 krb5kdc[21452](info): set up 2 sockets
Sep 26 09:36:37 dist-test-slave-21h8 krb5kdc[21452](info): commencing operation
krb5kdc: starting...
W20260926 09:36:38.883610 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:38 dist-test-slave-21h8 krb5kdc[21452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:38.994454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:39.073611 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:39.073832 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:39.073868 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:39.075112 21465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:39.075181 21465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:39.075204 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:39.075232 21465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:39.075263 21465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:39.076771 21465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:39.077025 21465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:39.077247 21465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:39.081876 21471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:39.081926 21470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:39.081995 21465 server_base.cc:1034] running on GCE node
W20260926 09:36:39.081911 21473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:39.082284 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:39.082511 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:39.083676 21465 hybrid_clock.cc:648] HybridClock initialized: now 1790415399083658 us; error 32 us; skew 500 ppm
Sep 26 09:36:39 dist-test-slave-21h8 krb5kdc[21452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:39.086048 21465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:39.086444 21465 webserver.cc:460] Webserver started at http://127.0.0.1:44517/ using document root <none> and password file <none>
I20260926 09:36:39.086670 21465 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:39.086753 21465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:39.086894 21465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:39.087961 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "8b0b75d682c44f6f83c38aa989920413"
format_stamp: "Formatted at 2026-09-26 09:36:39 on dist-test-slave-21h8"
server_key: "6d652f154bcb7562d3dc50c2335cd874"
server_key_iv: "eb223bd1027a0b96e51fccc0d7082de7"
server_key_version: "encryptionkey@0"
I20260926 09:36:39.088346 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b0b75d682c44f6f83c38aa989920413"
format_stamp: "Formatted at 2026-09-26 09:36:39 on dist-test-slave-21h8"
server_key: "6d652f154bcb7562d3dc50c2335cd874"
server_key_iv: "eb223bd1027a0b96e51fccc0d7082de7"
server_key_version: "encryptionkey@0"
I20260926 09:36:39.089591 21465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:36:39.090302 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:39.090461 21465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:39.090577 21465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "8b0b75d682c44f6f83c38aa989920413"
format_stamp: "Formatted at 2026-09-26 09:36:39 on dist-test-slave-21h8"
server_key: "6d652f154bcb7562d3dc50c2335cd874"
server_key_iv: "eb223bd1027a0b96e51fccc0d7082de7"
server_key_version: "encryptionkey@0"
I20260926 09:36:39.090653 21465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:39.094014 21465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:39.095786 21465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:39.095937 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:39.099581 21465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42683
I20260926 09:36:39.099629 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42683 every 8 connection(s)
I20260926 09:36:39.099937 21465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:39.100858 21533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:39.102953 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413: Bootstrap starting.
I20260926 09:36:39.103554 21533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:39.103829 21533 log.cc:824] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:39.104492 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413: No bootstrap required, opened a new log
I20260926 09:36:39.105995 21533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0b75d682c44f6f83c38aa989920413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260926 09:36:39.106122 21533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:39.106145 21533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0b75d682c44f6f83c38aa989920413, State: Initialized, Role: FOLLOWER
I20260926 09:36:39.106304 21533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0b75d682c44f6f83c38aa989920413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260926 09:36:39.106386 21533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:39.106429 21533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:39.106478 21533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:39.106985 21533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0b75d682c44f6f83c38aa989920413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260926 09:36:39.107096 21533 leader_election.cc:302] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b0b75d682c44f6f83c38aa989920413; no voters:
I20260926 09:36:39.107255 21533 leader_election.cc:288] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:39.107301 21536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:39.107501 21536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [term 1 LEADER]: Becoming Leader. State: Replica: 8b0b75d682c44f6f83c38aa989920413, State: Running, Role: LEADER
I20260926 09:36:39.107590 21533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:39.107626 21536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0b75d682c44f6f83c38aa989920413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260926 09:36:39.108057 21538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b0b75d682c44f6f83c38aa989920413. Latest consensus state: current_term: 1 leader_uuid: "8b0b75d682c44f6f83c38aa989920413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0b75d682c44f6f83c38aa989920413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260926 09:36:39.108153 21538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:39.108158 21537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b0b75d682c44f6f83c38aa989920413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0b75d682c44f6f83c38aa989920413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260926 09:36:39.108271 21537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:39.108430 21545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:39.108561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21465
I20260926 09:36:39.108665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:39.108866 392 external_mini_cluster.cc:1428] Setting key 474f053f61e15f48f9f67ae81976f25e
I20260926 09:36:39.108982 21545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:39.110458 21545 catalog_manager.cc:1347] Generated new cluster ID: 7f87c21d634b49289b58caa09e07c597
I20260926 09:36:39.110507 21545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:39.111784 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:39.109472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:40291 (local address 127.0.0.1:42683)
0926 09:36:39.109950 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:39.109958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:39.109989 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:36:39.110241 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:39.110248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:39.110276 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:39.110705 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:39.111191 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:39.111194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:39.111194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:39.111198 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:39.111229 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:39.111242 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:39.111532 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:39.111565 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 09:36:39.111608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260926 09:36:39.112748 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:39.120985 21545 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:39.121158 21545 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:39.121773 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:39.115710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:33995 (local address 127.0.0.1:42683)
0926 09:36:39.115745 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:39.115747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:39.120783 (+ 5036us) server_negotiation.cc:407] Connection header received
0926 09:36:39.120878 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:39.120878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:39.120883 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:39.120952 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:39.121562 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:39.121563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:39.121563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:39.121567 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:39.121581 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:39.121585 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:39.121641 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:39.121702 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:36:39.121715 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:39.127153 21545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b0b75d682c44f6f83c38aa989920413: Generated new TSK 0
I20260926 09:36:39.127349 21545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:39.128829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21465
2026-09-26T09:36:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:39 dist-test-slave-21h8 krb5kdc[21564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:39 dist-test-slave-21h8 krb5kdc[21564](info): set up 2 sockets
Sep 26 09:36:39 dist-test-slave-21h8 krb5kdc[21564](info): commencing operation
krb5kdc: starting...
W20260926 09:36:40.159601 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:40 dist-test-slave-21h8 krb5kdc[21564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:40.277693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:40.357174 21577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:40.357399 21577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:40.357438 21577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:40.358573 21577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:40.358645 21577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:40.358670 21577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:40.358700 21577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:40.358731 21577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:40.360221 21577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:40.360495 21577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:40.360715 21577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:40.365480 21582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:40.365506 21583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:40.365506 21585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:40.365950 21577 server_base.cc:1034] running on GCE node
I20260926 09:36:40.366209 21577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:40.366477 21577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:40.367635 21577 hybrid_clock.cc:648] HybridClock initialized: now 1790415400367615 us; error 49 us; skew 500 ppm
Sep 26 09:36:40 dist-test-slave-21h8 krb5kdc[21564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:40.373780 21577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:40.374262 21577 webserver.cc:460] Webserver started at http://127.0.0.1:42737/ using document root <none> and password file <none>
I20260926 09:36:40.374495 21577 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:40.374567 21577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:40.374716 21577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:40.375823 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "2357bd6477754ee684b19fc9efe14fe9"
format_stamp: "Formatted at 2026-09-26 09:36:40 on dist-test-slave-21h8"
server_key: "3b98f769edcbd4e39739eafab37b4e14"
server_key_iv: "fcba884f1e850dc3c3e2c290c3674b88"
server_key_version: "encryptionkey@0"
I20260926 09:36:40.376183 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "2357bd6477754ee684b19fc9efe14fe9"
format_stamp: "Formatted at 2026-09-26 09:36:40 on dist-test-slave-21h8"
server_key: "3b98f769edcbd4e39739eafab37b4e14"
server_key_iv: "fcba884f1e850dc3c3e2c290c3674b88"
server_key_version: "encryptionkey@0"
I20260926 09:36:40.377470 21577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:36:40.378219 21592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:40.378407 21577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:40.378515 21577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "2357bd6477754ee684b19fc9efe14fe9"
format_stamp: "Formatted at 2026-09-26 09:36:40 on dist-test-slave-21h8"
server_key: "3b98f769edcbd4e39739eafab37b4e14"
server_key_iv: "fcba884f1e850dc3c3e2c290c3674b88"
server_key_version: "encryptionkey@0"
I20260926 09:36:40.378592 21577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:40.390082 21577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:40.391839 21577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:40.391995 21577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:40.396092 21577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42765
I20260926 09:36:40.396129 21644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42765 every 8 connection(s)
I20260926 09:36:40.396450 21577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:40.397351 21645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:40.399741 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9: Bootstrap starting.
I20260926 09:36:40.400250 21645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:40.400471 21645 log.cc:824] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:40.401094 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9: No bootstrap required, opened a new log
I20260926 09:36:40.402426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21577
I20260926 09:36:40.402522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:40.402710 392 external_mini_cluster.cc:1428] Setting key 11b2dd43c7e1fec9bd13c0d09951643e
I20260926 09:36:40.402725 21645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357bd6477754ee684b19fc9efe14fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260926 09:36:40.402961 21645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:40.403004 21645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2357bd6477754ee684b19fc9efe14fe9, State: Initialized, Role: FOLLOWER
I20260926 09:36:40.403185 21645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357bd6477754ee684b19fc9efe14fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260926 09:36:40.403272 21645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:40.403360 21645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:40.403455 21645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:40.404484 21645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357bd6477754ee684b19fc9efe14fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260926 09:36:40.404588 21645 leader_election.cc:302] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2357bd6477754ee684b19fc9efe14fe9; no voters:
I20260926 09:36:40.404732 21645 leader_election.cc:288] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:40.404795 21650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:40.404939 21650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 2357bd6477754ee684b19fc9efe14fe9, State: Running, Role: LEADER
I20260926 09:36:40.404982 21645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:40.405088 21650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357bd6477754ee684b19fc9efe14fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260926 09:36:40.405671 21650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2357bd6477754ee684b19fc9efe14fe9. Latest consensus state: current_term: 1 leader_uuid: "2357bd6477754ee684b19fc9efe14fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357bd6477754ee684b19fc9efe14fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } } }
I20260926 09:36:40.405680 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2357bd6477754ee684b19fc9efe14fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2357bd6477754ee684b19fc9efe14fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } } }
I20260926 09:36:40.405771 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:40.405920 21650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:40.406054 21659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:40.406555 21659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:40.408048 21667 mvcc.cc:205] Tried to move back new op lower bound from 7333541480069206016 to 7333541480060485632. Current Snapshot: MvccSnapshot[applied={T|T < 7333541480069206016}]
I20260926 09:36:40.408669 21659 catalog_manager.cc:1347] Generated new cluster ID: ebe2c3b852174ab8ac67b8b3b5815914
I20260926 09:36:40.408723 21659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:36:40 dist-test-slave-21h8 krb5kdc[21564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:36:40.413381 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:40.403211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:59465 (local address 127.0.0.1:42765)
0926 09:36:40.403432 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:40.403441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:40.403473 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:36:40.403585 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:40.403592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:40.403648 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:40.403790 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:40.404369 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:40.405767 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:40.406955 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:40.407486 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:40.407937 (+ 451us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:40.410065 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:40.410069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:40.410070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:40.410081 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:40.411868 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:40.412251 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:40.412254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:40.412254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:40.412353 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:40.412590 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:40.412591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:40.412592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:40.412818 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:40.412995 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:40.413100 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 09:36:40.413209 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":176,"threads_started":1}
I20260926 09:36:40.414283 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:40.425933 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:40.415568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:55907 (local address 127.0.0.1:42765)
0926 09:36:40.415611 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:40.415613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:40.419490 (+ 3877us) server_negotiation.cc:407] Connection header received
0926 09:36:40.419564 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:40.419565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:40.419575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:40.419619 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:40.420281 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:40.421080 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:40.421767 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:40.421997 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:40.422072 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:40.423263 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:40.423264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:40.423264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:40.423272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:40.424740 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:40.425115 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:40.425116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:40.425117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:40.425175 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:40.425396 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:40.425397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:40.425398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:40.425514 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:40.425635 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:40.425749 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 09:36:40.425810 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260926 09:36:40.451468 21659 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:40.452014 21659 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:40.454419 21659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2357bd6477754ee684b19fc9efe14fe9: Generated new TSK 0
I20260926 09:36:40.454622 21659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:40.458774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21577
W20260926 09:36:40.462570 21562 connection.cc:535] client connection to 127.0.0.1:42765 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42765): Connection reset by peer (error 104)
2026-09-26T09:36:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:40 dist-test-slave-21h8 krb5kdc[21676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:40 dist-test-slave-21h8 krb5kdc[21676](info): set up 2 sockets
Sep 26 09:36:40 dist-test-slave-21h8 krb5kdc[21676](info): commencing operation
krb5kdc: starting...
W20260926 09:36:41.491406 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:41 dist-test-slave-21h8 krb5kdc[21676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:41.602321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:41.680699 21689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:41.680934 21689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:41.680969 21689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:41.682147 21689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:41.682215 21689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:41.682238 21689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:41.682261 21689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:41.682289 21689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:41.683760 21689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:41.684039 21689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:41.684259 21689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:41.688966 21695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:41.689011 21694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:41.689181 21689 server_base.cc:1034] running on GCE node
W20260926 09:36:41.688967 21697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:41.689482 21689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:41.689770 21689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:41.690950 21689 hybrid_clock.cc:648] HybridClock initialized: now 1790415401690943 us; error 48 us; skew 500 ppm
Sep 26 09:36:41 dist-test-slave-21h8 krb5kdc[21676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:41.693467 21689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:41.693917 21689 webserver.cc:460] Webserver started at http://127.0.0.1:32827/ using document root <none> and password file <none>
I20260926 09:36:41.694160 21689 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:41.694226 21689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:41.694373 21689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:41.695441 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "89b33d9d428348a3867d27e61118d6e9"
format_stamp: "Formatted at 2026-09-26 09:36:41 on dist-test-slave-21h8"
server_key: "b0c1ad2adf542b15bdb11af642b5207e"
server_key_iv: "32a9762ca9454efa7bdd18fe323db36a"
server_key_version: "encryptionkey@0"
I20260926 09:36:41.695782 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "89b33d9d428348a3867d27e61118d6e9"
format_stamp: "Formatted at 2026-09-26 09:36:41 on dist-test-slave-21h8"
server_key: "b0c1ad2adf542b15bdb11af642b5207e"
server_key_iv: "32a9762ca9454efa7bdd18fe323db36a"
server_key_version: "encryptionkey@0"
I20260926 09:36:41.697072 21689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:36:41.697794 21704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:41.697962 21689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:41.698067 21689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "89b33d9d428348a3867d27e61118d6e9"
format_stamp: "Formatted at 2026-09-26 09:36:41 on dist-test-slave-21h8"
server_key: "b0c1ad2adf542b15bdb11af642b5207e"
server_key_iv: "32a9762ca9454efa7bdd18fe323db36a"
server_key_version: "encryptionkey@0"
I20260926 09:36:41.698137 21689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:41.719832 21689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:41.722163 21689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:41.722328 21689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:41.726496 21689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45729
I20260926 09:36:41.726544 21756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45729 every 8 connection(s)
I20260926 09:36:41.726864 21689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:41.727849 21757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:41.730196 21757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9: Bootstrap starting.
I20260926 09:36:41.730813 21757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:41.731114 21757 log.cc:824] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:41.731882 21757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9: No bootstrap required, opened a new log
I20260926 09:36:41.733470 21757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b33d9d428348a3867d27e61118d6e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45729 } }
I20260926 09:36:41.733639 21757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:41.733693 21757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b33d9d428348a3867d27e61118d6e9, State: Initialized, Role: FOLLOWER
I20260926 09:36:41.733855 21757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b33d9d428348a3867d27e61118d6e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45729 } }
I20260926 09:36:41.733942 21757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:41.733984 21757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:41.734037 21757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:41.734579 21757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b33d9d428348a3867d27e61118d6e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45729 } }
I20260926 09:36:41.734701 21757 leader_election.cc:302] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b33d9d428348a3867d27e61118d6e9; no voters:
I20260926 09:36:41.734900 21757 leader_election.cc:288] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:41.734925 21760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:41.735106 21760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [term 1 LEADER]: Becoming Leader. State: Replica: 89b33d9d428348a3867d27e61118d6e9, State: Running, Role: LEADER
I20260926 09:36:41.735169 21757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:41.735247 21760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b33d9d428348a3867d27e61118d6e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45729 } }
I20260926 09:36:41.735736 21762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b33d9d428348a3867d27e61118d6e9. Latest consensus state: current_term: 1 leader_uuid: "89b33d9d428348a3867d27e61118d6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b33d9d428348a3867d27e61118d6e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45729 } } }
I20260926 09:36:41.735821 21762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:41.735854 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89b33d9d428348a3867d27e61118d6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b33d9d428348a3867d27e61118d6e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45729 } } }
I20260926 09:36:41.735946 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:41.736174 21769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:41.736596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21689
I20260926 09:36:41.736688 21769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:41.736711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:41.736937 392 external_mini_cluster.cc:1428] Setting key 9aeb8700f57e013f979b30dc689f0a54
I20260926 09:36:41.738344 21769 catalog_manager.cc:1347] Generated new cluster ID: 0cad78d68c9647409d2f776493494c81
I20260926 09:36:41.738399 21769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:41.742851 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:41.737553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:38983 (local address 127.0.0.1:45729)
0926 09:36:41.737730 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:41.737738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:41.737771 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:36:41.737902 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:41.737908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:41.737982 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:41.738129 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:41.738394 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:41.739746 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:41.740787 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:41.741341 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:41.741448 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:41.742169 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:41.742172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:41.742173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:41.742362 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:41.742396 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:41.742409 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:41.742495 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:41.742608 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 09:36:41.742632 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260926 09:36:41.743800 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:41.750187 21769 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:41.750711 21769 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:41.753495 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:41.745348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:55183 (local address 127.0.0.1:45729)
0926 09:36:41.745401 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:41.745403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:41.749916 (+ 4513us) server_negotiation.cc:407] Connection header received
0926 09:36:41.750026 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:41.750027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:41.750042 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:41.750116 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:41.750361 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:41.751116 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:41.752118 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:41.752444 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:41.752523 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:41.753205 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:41.753207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:41.753207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:41.753212 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:41.753227 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:41.753232 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:41.753312 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:41.753407 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 09:36:41.753421 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:36:41.756494 21769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89b33d9d428348a3867d27e61118d6e9: Generated new TSK 0
I20260926 09:36:41.756657 21769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:41.759951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21689
W20260926 09:36:41.763840 21674 connection.cc:535] client connection to 127.0.0.1:45729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45729): Connection reset by peer (error 104)
2026-09-26T09:36:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:41 dist-test-slave-21h8 krb5kdc[21788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:41 dist-test-slave-21h8 krb5kdc[21788](info): set up 2 sockets
Sep 26 09:36:41 dist-test-slave-21h8 krb5kdc[21788](info): commencing operation
krb5kdc: starting...
W20260926 09:36:42.795511 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:42 dist-test-slave-21h8 krb5kdc[21788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:36:42.913986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:42.992165 21801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:42.992354 21801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:42.992377 21801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:42.993362 21801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:42.993393 21801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:42.993407 21801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:42.993417 21801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:42.993448 21801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:42.994789 21801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:42.994982 21801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:42.995128 21801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:36:42.999786 21801 server_base.cc:1034] running on GCE node
W20260926 09:36:42.999755 21806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:42.999759 21807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:42.999742 21809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:43.000268 21801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:43.000584 21801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:43.001744 21801 hybrid_clock.cc:648] HybridClock initialized: now 1790415403001740 us; error 35 us; skew 500 ppm
Sep 26 09:36:43 dist-test-slave-21h8 krb5kdc[21788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:43.004153 21801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:36:43.004601 21801 webserver.cc:460] Webserver started at http://127.0.0.1:35247/ using document root <none> and password file <none>
I20260926 09:36:43.004822 21801 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:43.004889 21801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:43.005017 21801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:43.006031 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4"
format_stamp: "Formatted at 2026-09-26 09:36:43 on dist-test-slave-21h8"
server_key: "2be39104d0327b78ef4c7c88c6116d31"
server_key_iv: "4a2192c4786dde53decc6686adbe12b3"
server_key_version: "encryptionkey@0"
I20260926 09:36:43.006379 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4"
format_stamp: "Formatted at 2026-09-26 09:36:43 on dist-test-slave-21h8"
server_key: "2be39104d0327b78ef4c7c88c6116d31"
server_key_iv: "4a2192c4786dde53decc6686adbe12b3"
server_key_version: "encryptionkey@0"
I20260926 09:36:43.007688 21801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:43.008354 21816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:43.008514 21801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:43.008602 21801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4"
format_stamp: "Formatted at 2026-09-26 09:36:43 on dist-test-slave-21h8"
server_key: "2be39104d0327b78ef4c7c88c6116d31"
server_key_iv: "4a2192c4786dde53decc6686adbe12b3"
server_key_version: "encryptionkey@0"
I20260926 09:36:43.008652 21801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:43.011801 21801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:43.013494 21801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:43.013590 21801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:43.017323 21801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35789
I20260926 09:36:43.017346 21868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35789 every 8 connection(s)
I20260926 09:36:43.017627 21801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:43.018493 21869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:43.018621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21801
I20260926 09:36:43.018739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:43.018911 392 external_mini_cluster.cc:1428] Setting key 01c9bb2efa185152c56656a2ec3b471b
I20260926 09:36:43.021237 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:43.019408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:56325 (local address 127.0.0.1:35789)
0926 09:36:43.019583 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:43.019590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:43.019629 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:36:43.020056 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:43.020061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:43.020085 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:43.020189 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:43.020818 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:43.020821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:43.020822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:43.020825 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:43.020859 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:43.020869 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:43.020919 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:43.021005 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 09:36:43.021034 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260926 09:36:43.021879 21869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4: Bootstrap starting.
I20260926 09:36:43.022181 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:43.022584 21869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:43.022904 21869 log.cc:824] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:43.023658 21869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4: No bootstrap required, opened a new log
I20260926 09:36:43.024767 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:43.023771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:46871 (local address 127.0.0.1:35789)
0926 09:36:43.023814 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:43.023816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:43.023967 (+ 151us) server_negotiation.cc:407] Connection header received
0926 09:36:43.024021 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:43.024022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:43.024025 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:43.024065 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:43.024570 (+ 505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:43.024570 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:43.024571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:43.024574 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:43.024583 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:43.024586 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:43.024633 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:43.024682 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 09:36:43.024691 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:36:43.026001 21869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } }
I20260926 09:36:43.026127 21869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:43.026149 21869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f12ef2baca4b0eb72ff3fff8ff2ce4, State: Initialized, Role: FOLLOWER
I20260926 09:36:43.026289 21869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } }
I20260926 09:36:43.026369 21869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:43.026407 21869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:43.026456 21869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:43.026928 21869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } }
I20260926 09:36:43.027042 21869 leader_election.cc:302] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06f12ef2baca4b0eb72ff3fff8ff2ce4; no voters:
I20260926 09:36:43.027215 21869 leader_election.cc:288] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:43.027261 21879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:43.027460 21869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:43.027504 21879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 06f12ef2baca4b0eb72ff3fff8ff2ce4, State: Running, Role: LEADER
I20260926 09:36:43.027607 21879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } }
I20260926 09:36:43.027894 21879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06f12ef2baca4b0eb72ff3fff8ff2ce4. Latest consensus state: current_term: 1 leader_uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } } }
I20260926 09:36:43.028012 21879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:43.027904 21881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f12ef2baca4b0eb72ff3fff8ff2ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } } }
I20260926 09:36:43.028119 21881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:43.028317 21888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:43.028791 21888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:43.030184 21888 catalog_manager.cc:1347] Generated new cluster ID: a82850a5ced348b79a6977ba53000562
I20260926 09:36:43.030231 21888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:43.046823 21888 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:43.046968 21888 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:43.051326 21888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06f12ef2baca4b0eb72ff3fff8ff2ce4: Generated new TSK 0
I20260926 09:36:43.051498 21888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:43.054675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21801
2026-09-26T09:36:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:43 dist-test-slave-21h8 krb5kdc[21900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:43 dist-test-slave-21h8 krb5kdc[21900](info): set up 2 sockets
Sep 26 09:36:43 dist-test-slave-21h8 krb5kdc[21900](info): commencing operation
krb5kdc: starting...
W20260926 09:36:44.091504 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:44 dist-test-slave-21h8 krb5kdc[21900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:44.202785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41471
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:44.282248 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:44.282454 21913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:44.282476 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:44.283659 21913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:44.283699 21913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:44.283712 21913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:44.283751 21913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:44.283782 21913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:44.285199 21913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:44.285444 21913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:44.285668 21913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:44.290416 21918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:44.290582 21913 server_base.cc:1034] running on GCE node
W20260926 09:36:44.290424 21921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:44.290421 21919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:44.290931 21913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:44.291263 21913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:44.292495 21913 hybrid_clock.cc:648] HybridClock initialized: now 1790415404292474 us; error 39 us; skew 500 ppm
Sep 26 09:36:44 dist-test-slave-21h8 krb5kdc[21900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:44.295217 21913 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:44.295732 21913 webserver.cc:460] Webserver started at http://127.0.98.62:36961/ using document root <none> and password file <none>
I20260926 09:36:44.295977 21913 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:44.296051 21913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:44.296200 21913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:44.297260 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "e17bde3ed238494d81a3105ecc5386b6"
format_stamp: "Formatted at 2026-09-26 09:36:44 on dist-test-slave-21h8"
server_key: "824139b586a0523c0979c9a5368a81ce"
server_key_iv: "654b889f44c5a9cd9f832d322e662db2"
server_key_version: "encryptionkey@0"
I20260926 09:36:44.297626 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "e17bde3ed238494d81a3105ecc5386b6"
format_stamp: "Formatted at 2026-09-26 09:36:44 on dist-test-slave-21h8"
server_key: "824139b586a0523c0979c9a5368a81ce"
server_key_iv: "654b889f44c5a9cd9f832d322e662db2"
server_key_version: "encryptionkey@0"
I20260926 09:36:44.298947 21913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:36:44.299803 21928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:44.299975 21913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:44.300078 21913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "e17bde3ed238494d81a3105ecc5386b6"
format_stamp: "Formatted at 2026-09-26 09:36:44 on dist-test-slave-21h8"
server_key: "824139b586a0523c0979c9a5368a81ce"
server_key_iv: "654b889f44c5a9cd9f832d322e662db2"
server_key_version: "encryptionkey@0"
I20260926 09:36:44.300164 21913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:44.330231 21913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:44.332405 21913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:44.332581 21913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:44.336704 21913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41471
I20260926 09:36:44.336743 21980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41471 every 8 connection(s)
I20260926 09:36:44.337117 21913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:44.337626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 21913
I20260926 09:36:44.337728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:44.337877 392 external_mini_cluster.cc:1428] Setting key a86b139fac8a78162353e38f1ca0abe4
I20260926 09:36:44.338714 21981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:44.341600 21981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6: Bootstrap starting.
I20260926 09:36:44.342288 21981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:44.342600 21981 log.cc:824] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:44.343415 21981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6: No bootstrap required, opened a new log
Sep 26 09:36:44 dist-test-slave-21h8 krb5kdc[21900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:44.345858 21981 raft_consensus.cc:348] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17bde3ed238494d81a3105ecc5386b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260926 09:36:44.346025 21981 raft_consensus.cc:374] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:44.346053 21981 raft_consensus.cc:729] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17bde3ed238494d81a3105ecc5386b6, State: Initialized, Role: FOLLOWER
I20260926 09:36:44.346206 21981 consensus_queue.cc:260] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17bde3ed238494d81a3105ecc5386b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260926 09:36:44.346308 21981 raft_consensus.cc:388] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:44.346343 21981 raft_consensus.cc:482] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:44.346396 21981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:44.346926 21981 raft_consensus.cc:504] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17bde3ed238494d81a3105ecc5386b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260926 09:36:44.347052 21981 leader_election.cc:302] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e17bde3ed238494d81a3105ecc5386b6; no voters:
I20260926 09:36:44.347229 21981 leader_election.cc:288] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:44.347270 21986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:44.347456 21986 raft_consensus.cc:686] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [term 1 LEADER]: Becoming Leader. State: Replica: e17bde3ed238494d81a3105ecc5386b6, State: Running, Role: LEADER
I20260926 09:36:44.347589 21981 sys_catalog.cc:564] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:44.347577 21986 consensus_queue.cc:237] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17bde3ed238494d81a3105ecc5386b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260926 09:36:44.348037 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e17bde3ed238494d81a3105ecc5386b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17bde3ed238494d81a3105ecc5386b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } } }
I20260926 09:36:44.348110 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:44.348327 21988 sys_catalog.cc:455] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e17bde3ed238494d81a3105ecc5386b6. Latest consensus state: current_term: 1 leader_uuid: "e17bde3ed238494d81a3105ecc5386b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17bde3ed238494d81a3105ecc5386b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } } }
I20260926 09:36:44.348639 21988 sys_catalog.cc:458] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:44.348794 21992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:44.349200 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:44.338374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49381 (local address 127.0.98.62:41471)
0926 09:36:44.338583 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:44.338596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:44.338621 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:36:44.339112 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:44.339119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:44.339164 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:44.339275 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:44.340098 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:44.341001 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:44.342078 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:44.342525 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:44.342611 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:44.345400 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:44.345404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:44.345404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:44.345415 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:44.347193 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:44.347706 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:44.347710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:44.347711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:44.347810 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:44.348087 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:44.348088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:44.348088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:44.348444 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:44.348587 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:44.348882 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 09:36:44.348984 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
I20260926 09:36:44.349295 21992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:44.350510 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:44.350936 21992 catalog_manager.cc:1347] Generated new cluster ID: c64dd7f263ab4c48a4918d62a70150a3
I20260926 09:36:44.350992 21992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:44.358209 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:44.351977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48479 (local address 127.0.98.62:41471)
0926 09:36:44.352010 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:44.352013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:44.352146 (+ 133us) server_negotiation.cc:407] Connection header received
0926 09:36:44.352216 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:44.352217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:44.352226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:44.352280 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:44.352768 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:44.353256 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:44.353964 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:44.354199 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:44.354277 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:44.355562 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:44.355564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:44.355564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:44.355572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:44.357038 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:44.357422 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:44.357423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:44.357423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:44.357479 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:44.357701 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:44.357702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:44.357703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:44.357824 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:44.357934 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:44.358054 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 09:36:44.358114 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 09:36:44.375934 21992 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:44.376674 21992 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:44.382108 21992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e17bde3ed238494d81a3105ecc5386b6: Generated new TSK 0
I20260926 09:36:44.382302 21992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:44.388276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 21913
W20260926 09:36:44.392722 21898 connection.cc:535] client connection to 127.0.98.62:41471 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41471): Connection reset by peer (error 104)
2026-09-26T09:36:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:44 dist-test-slave-21h8 krb5kdc[22012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:44 dist-test-slave-21h8 krb5kdc[22012](info): set up 2 sockets
Sep 26 09:36:44 dist-test-slave-21h8 krb5kdc[22012](info): commencing operation
krb5kdc: starting...
W20260926 09:36:45.419229 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:45 dist-test-slave-21h8 krb5kdc[22012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:45.536125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:45.614629 22025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:45.614871 22025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:45.614907 22025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:45.616061 22025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:45.616135 22025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:45.616160 22025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:45.616189 22025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:45.616235 22025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:45.617697 22025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:45.617969 22025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:45.618186 22025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:45.623278 22031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:45.623273 22033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:45.623497 22025 server_base.cc:1034] running on GCE node
W20260926 09:36:45.623273 22030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:45.623800 22025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:45.624089 22025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:45.625275 22025 hybrid_clock.cc:648] HybridClock initialized: now 1790415405625270 us; error 38 us; skew 500 ppm
Sep 26 09:36:45 dist-test-slave-21h8 krb5kdc[22012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:45.635578 22025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:45.636092 22025 webserver.cc:460] Webserver started at http://127.0.98.62:43407/ using document root <none> and password file <none>
I20260926 09:36:45.636322 22025 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:45.636389 22025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:45.636548 22025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:45.637606 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "dd59ba739cb24273855da5ed65063f3a"
format_stamp: "Formatted at 2026-09-26 09:36:45 on dist-test-slave-21h8"
server_key: "baa3078c9004fe0e5a73917ba877c4a3"
server_key_iv: "96a49847e63ae412d921da6ba8154a34"
server_key_version: "encryptionkey@0"
I20260926 09:36:45.637964 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd59ba739cb24273855da5ed65063f3a"
format_stamp: "Formatted at 2026-09-26 09:36:45 on dist-test-slave-21h8"
server_key: "baa3078c9004fe0e5a73917ba877c4a3"
server_key_iv: "96a49847e63ae412d921da6ba8154a34"
server_key_version: "encryptionkey@0"
I20260926 09:36:45.639287 22025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 09:36:45.640113 22040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:45.640282 22025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:45.640389 22025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "dd59ba739cb24273855da5ed65063f3a"
format_stamp: "Formatted at 2026-09-26 09:36:45 on dist-test-slave-21h8"
server_key: "baa3078c9004fe0e5a73917ba877c4a3"
server_key_iv: "96a49847e63ae412d921da6ba8154a34"
server_key_version: "encryptionkey@0"
I20260926 09:36:45.640486 22025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:45.651976 22025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:45.654263 22025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:45.654440 22025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:45.658396 22025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39721
I20260926 09:36:45.658427 22092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39721 every 8 connection(s)
I20260926 09:36:45.658798 22025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:45.659674 22093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:45.660343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22025
I20260926 09:36:45.660454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:45.660633 392 external_mini_cluster.cc:1428] Setting key 90892da6ba2ed4247059bb51825dee89
I20260926 09:36:45.662568 22093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a: Bootstrap starting.
I20260926 09:36:45.663141 22093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:45.663483 22093 log.cc:824] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:45.664213 22093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a: No bootstrap required, opened a new log
I20260926 09:36:45.666146 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:45.661176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47893 (local address 127.0.98.62:39721)
0926 09:36:45.661360 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:45.661375 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:45.661399 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:36:45.661793 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:45.661799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:45.661842 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:45.661952 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:45.662216 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:45.663144 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:45.664205 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:45.664690 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:45.664771 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:45.665465 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:45.665469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:45.665469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:45.665671 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:45.665700 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:45.665710 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:45.665774 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:45.665903 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 09:36:45.665944 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260926 09:36:45.666473 22093 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd59ba739cb24273855da5ed65063f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39721 } }
I20260926 09:36:45.666636 22093 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:45.666730 22093 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd59ba739cb24273855da5ed65063f3a, State: Initialized, Role: FOLLOWER
I20260926 09:36:45.666888 22093 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd59ba739cb24273855da5ed65063f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39721 } }
I20260926 09:36:45.666978 22093 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:45.667011 22093 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:45.667094 22093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:45.667176 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:45.667683 22093 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd59ba739cb24273855da5ed65063f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39721 } }
I20260926 09:36:45.667798 22093 leader_election.cc:302] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd59ba739cb24273855da5ed65063f3a; no voters:
I20260926 09:36:45.667979 22093 leader_election.cc:288] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:45.668007 22098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:45.668125 22098 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [term 1 LEADER]: Becoming Leader. State: Replica: dd59ba739cb24273855da5ed65063f3a, State: Running, Role: LEADER
I20260926 09:36:45.668238 22093 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:45.668268 22098 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd59ba739cb24273855da5ed65063f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39721 } }
I20260926 09:36:45.668572 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd59ba739cb24273855da5ed65063f3a. Latest consensus state: current_term: 1 leader_uuid: "dd59ba739cb24273855da5ed65063f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd59ba739cb24273855da5ed65063f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39721 } } }
I20260926 09:36:45.668608 22099 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd59ba739cb24273855da5ed65063f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd59ba739cb24273855da5ed65063f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39721 } } }
I20260926 09:36:45.668704 22099 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:45.668619 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:45.668910 22106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:45.669378 22106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:45.671060 22106 catalog_manager.cc:1347] Generated new cluster ID: 6fdf38546bf34e5d9fd72925c32e0d60
I20260926 09:36:45.671118 22106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:45.673552 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:45.670341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44987 (local address 127.0.98.62:39721)
0926 09:36:45.670375 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:45.670377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:45.670387 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:45.670444 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:45.670444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:45.670473 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:45.670530 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:45.670721 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:45.671377 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:45.672360 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:45.672564 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:45.672680 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:45.673344 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:45.673345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:45.673345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:45.673350 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:45.673362 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:45.673366 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:45.673417 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:45.673474 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 09:36:45.673482 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:36:45.703361 22106 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:45.703938 22106 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:45.714855 22106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd59ba739cb24273855da5ed65063f3a: Generated new TSK 0
I20260926 09:36:45.715059 22106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:45.721909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22025
W20260926 09:36:45.726219 22010 connection.cc:535] client connection to 127.0.98.62:39721 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39721): Connection reset by peer (error 104)
2026-09-26T09:36:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:45 dist-test-slave-21h8 krb5kdc[22124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:45 dist-test-slave-21h8 krb5kdc[22124](info): set up 2 sockets
Sep 26 09:36:45 dist-test-slave-21h8 krb5kdc[22124](info): commencing operation
krb5kdc: starting...
W20260926 09:36:46.763597 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:46 dist-test-slave-21h8 krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:46.879796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:46.958885 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:46.959091 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:46.959118 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:46.960310 22137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:46.960352 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:46.960366 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:46.960381 22137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:46.960392 22137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:46.961885 22137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:46.962137 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:46.962359 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:46.967046 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:46.967022 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:46.967028 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:46.967350 22137 server_base.cc:1034] running on GCE node
I20260926 09:36:46.967589 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:46.967851 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:46.969038 22137 hybrid_clock.cc:648] HybridClock initialized: now 1790415406969023 us; error 29 us; skew 500 ppm
Sep 26 09:36:46 dist-test-slave-21h8 krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:46.971612 22137 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:46.972016 22137 webserver.cc:460] Webserver started at http://127.0.98.62:41875/ using document root <none> and password file <none>
I20260926 09:36:46.972208 22137 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:46.972251 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:46.972348 22137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:46.973417 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "9f8c630f09184c83be063f8f2464e848"
format_stamp: "Formatted at 2026-09-26 09:36:46 on dist-test-slave-21h8"
server_key: "be2537d21188c6dcc4f95e0969b49eae"
server_key_iv: "71d0ee81323f5263699bf109c10a668f"
server_key_version: "encryptionkey@0"
I20260926 09:36:46.973722 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f8c630f09184c83be063f8f2464e848"
format_stamp: "Formatted at 2026-09-26 09:36:46 on dist-test-slave-21h8"
server_key: "be2537d21188c6dcc4f95e0969b49eae"
server_key_iv: "71d0ee81323f5263699bf109c10a668f"
server_key_version: "encryptionkey@0"
I20260926 09:36:46.975016 22137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:36:46.975803 22152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:46.975963 22137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:46.976045 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "9f8c630f09184c83be063f8f2464e848"
format_stamp: "Formatted at 2026-09-26 09:36:46 on dist-test-slave-21h8"
server_key: "be2537d21188c6dcc4f95e0969b49eae"
server_key_iv: "71d0ee81323f5263699bf109c10a668f"
server_key_version: "encryptionkey@0"
I20260926 09:36:46.976099 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:46.979126 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:46.980890 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:46.980997 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:46.984609 22137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39913
I20260926 09:36:46.984660 22204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39913 every 8 connection(s)
I20260926 09:36:46.984974 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:46.985970 22205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:46.988329 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848: Bootstrap starting.
I20260926 09:36:46.988899 22205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:46.989193 22205 log.cc:824] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:46.989873 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848: No bootstrap required, opened a new log
I20260926 09:36:46.991678 22205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8c630f09184c83be063f8f2464e848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } }
I20260926 09:36:46.991853 22205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:46.991905 22205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8c630f09184c83be063f8f2464e848, State: Initialized, Role: FOLLOWER
I20260926 09:36:46.992070 22205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8c630f09184c83be063f8f2464e848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } }
I20260926 09:36:46.992177 22205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:46.992218 22205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:46.992278 22205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:46.992825 22205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8c630f09184c83be063f8f2464e848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } }
I20260926 09:36:46.992944 22205 leader_election.cc:302] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f8c630f09184c83be063f8f2464e848; no voters:
I20260926 09:36:46.993146 22205 leader_election.cc:288] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:46.993186 22208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:46.993345 22208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [term 1 LEADER]: Becoming Leader. State: Replica: 9f8c630f09184c83be063f8f2464e848, State: Running, Role: LEADER
I20260926 09:36:46.993417 22205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:46.993490 22208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8c630f09184c83be063f8f2464e848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } }
I20260926 09:36:46.993901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22137
I20260926 09:36:46.993922 22209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f8c630f09184c83be063f8f2464e848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8c630f09184c83be063f8f2464e848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } } }
I20260926 09:36:46.994031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:46.994029 22209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:46.993930 22210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8c630f09184c83be063f8f2464e848. Latest consensus state: current_term: 1 leader_uuid: "9f8c630f09184c83be063f8f2464e848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8c630f09184c83be063f8f2464e848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } } }
I20260926 09:36:46.994127 22210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:46.994241 392 external_mini_cluster.cc:1428] Setting key 940f1df83ba2ecf6eed37423439eb484
I20260926 09:36:46.994510 22212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:46.995085 22212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:46.996655 22212 catalog_manager.cc:1347] Generated new cluster ID: bc6bbd2b2d6e40278e6eca84769a5ca2
I20260926 09:36:46.996712 22212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:46.997066 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:46.994865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51847 (local address 127.0.98.62:39913)
0926 09:36:46.995089 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:46.995101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:46.995385 (+ 284us) server_negotiation.cc:407] Connection header received
0926 09:36:46.995714 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:46.995723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:46.995762 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:46.995912 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:46.996621 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:46.996624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:46.996624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:46.996629 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:46.996667 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:46.996683 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:46.996740 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:46.996846 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 09:36:46.996892 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260926 09:36:46.998167 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:47.004498 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:46.999816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56175 (local address 127.0.98.62:39913)
0926 09:36:46.999854 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:46.999856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:47.003511 (+ 3655us) server_negotiation.cc:407] Connection header received
0926 09:36:47.003605 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:47.003606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:47.003613 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:47.003646 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:47.004270 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:47.004271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:47.004271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:47.004275 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:47.004289 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:47.004293 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:47.004346 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:47.004424 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 09:36:47.004441 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:47.010205 22212 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:47.010367 22212 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:47.015697 22212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f8c630f09184c83be063f8f2464e848: Generated new TSK 0
I20260926 09:36:47.015864 22212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:47.019219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22137
2026-09-26T09:36:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:47 dist-test-slave-21h8 krb5kdc[22236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:47 dist-test-slave-21h8 krb5kdc[22236](info): set up 2 sockets
Sep 26 09:36:47 dist-test-slave-21h8 krb5kdc[22236](info): commencing operation
krb5kdc: starting...
W20260926 09:36:48.099273 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:48 dist-test-slave-21h8 krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:48.212266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42751
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:48.292608 22249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:48.292795 22249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:48.292817 22249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:48.293855 22249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:48.293890 22249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:48.293905 22249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:48.293917 22249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:48.293936 22249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:48.295274 22249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:48.295481 22249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:48.295641 22249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:48.300554 22257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:48.300555 22254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:48.300848 22249 server_base.cc:1034] running on GCE node
W20260926 09:36:48.300550 22255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:48.301151 22249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:48.301488 22249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:48.302649 22249 hybrid_clock.cc:648] HybridClock initialized: now 1790415408302630 us; error 50 us; skew 500 ppm
Sep 26 09:36:48 dist-test-slave-21h8 krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:48.305271 22249 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:48.305744 22249 webserver.cc:460] Webserver started at http://127.0.98.62:34585/ using document root <none> and password file <none>
I20260926 09:36:48.305968 22249 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:48.306054 22249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:48.306203 22249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:48.307245 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "ec8c0efbd31a4cb58df8b8c71d911576"
format_stamp: "Formatted at 2026-09-26 09:36:48 on dist-test-slave-21h8"
server_key: "1ee7a4266a183263db1542b1406ec0b5"
server_key_iv: "e41a1b7a349bc7a773a9532c3b35202f"
server_key_version: "encryptionkey@0"
I20260926 09:36:48.307663 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec8c0efbd31a4cb58df8b8c71d911576"
format_stamp: "Formatted at 2026-09-26 09:36:48 on dist-test-slave-21h8"
server_key: "1ee7a4266a183263db1542b1406ec0b5"
server_key_iv: "e41a1b7a349bc7a773a9532c3b35202f"
server_key_version: "encryptionkey@0"
I20260926 09:36:48.308970 22249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:48.309762 22264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:48.309949 22249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:48.310048 22249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "ec8c0efbd31a4cb58df8b8c71d911576"
format_stamp: "Formatted at 2026-09-26 09:36:48 on dist-test-slave-21h8"
server_key: "1ee7a4266a183263db1542b1406ec0b5"
server_key_iv: "e41a1b7a349bc7a773a9532c3b35202f"
server_key_version: "encryptionkey@0"
I20260926 09:36:48.310134 22249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:48.331387 22249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:48.333695 22249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:48.333871 22249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:48.338362 22249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42751
I20260926 09:36:48.338424 22316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42751 every 8 connection(s)
I20260926 09:36:48.338791 22249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:48.339722 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:48.342221 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576: Bootstrap starting.
I20260926 09:36:48.342861 22317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:48.343199 22317 log.cc:824] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:48.343955 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576: No bootstrap required, opened a new log
I20260926 09:36:48.345747 22317 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8c0efbd31a4cb58df8b8c71d911576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42751 } }
I20260926 09:36:48.345898 22317 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:48.345927 22317 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c0efbd31a4cb58df8b8c71d911576, State: Initialized, Role: FOLLOWER
I20260926 09:36:48.346125 22317 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8c0efbd31a4cb58df8b8c71d911576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42751 } }
I20260926 09:36:48.346207 22317 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:48.346256 22317 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:48.346319 22317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:48.346823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22249
I20260926 09:36:48.346947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:48.347113 22317 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8c0efbd31a4cb58df8b8c71d911576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42751 } }
I20260926 09:36:48.347203 392 external_mini_cluster.cc:1428] Setting key 34cd8e0c40321849f13f689b6a44ea9f
I20260926 09:36:48.347249 22317 leader_election.cc:302] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec8c0efbd31a4cb58df8b8c71d911576; no voters:
I20260926 09:36:48.347463 22317 leader_election.cc:288] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:48.347499 22320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:48.347659 22320 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [term 1 LEADER]: Becoming Leader. State: Replica: ec8c0efbd31a4cb58df8b8c71d911576, State: Running, Role: LEADER
I20260926 09:36:48.347761 22317 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:48.347826 22320 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8c0efbd31a4cb58df8b8c71d911576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42751 } }
I20260926 09:36:48.348147 22322 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec8c0efbd31a4cb58df8b8c71d911576. Latest consensus state: current_term: 1 leader_uuid: "ec8c0efbd31a4cb58df8b8c71d911576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8c0efbd31a4cb58df8b8c71d911576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42751 } } }
I20260926 09:36:48.348716 22322 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:48.348176 22321 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec8c0efbd31a4cb58df8b8c71d911576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8c0efbd31a4cb58df8b8c71d911576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42751 } } }
I20260926 09:36:48.348937 22321 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:48.349200 22332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:48.349752 22332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:48.351297 22332 catalog_manager.cc:1347] Generated new cluster ID: e35510d7d23c4eb7bfb97dc460644296
I20260926 09:36:48.352404 22332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:36:48 dist-test-slave-21h8 krb5kdc[22236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:48.361156 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:48.347910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35815 (local address 127.0.98.62:42751)
0926 09:36:48.348334 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:48.348343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:48.348375 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:36:48.348489 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:48.348496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:48.348550 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:48.349861 (+ 1311us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:48.350819 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:48.352117 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:48.353363 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:48.354016 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:48.354130 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:48.357241 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:48.357245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:48.357245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:48.357258 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:48.359391 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:48.359796 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:48.359799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:48.359799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:48.359940 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:48.360164 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:48.360165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:48.360165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:48.360496 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:48.360659 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:48.360770 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:36:48.360917 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":94,"threads_started":1}
I20260926 09:36:48.362192 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:48.364409 22332 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:48.365193 22332 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:48.371867 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:48.364113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60299 (local address 127.0.98.62:42751)
0926 09:36:48.364254 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:48.364256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:48.364271 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:36:48.364334 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:48.364335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:48.364347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:48.364417 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:48.365193 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:48.365944 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:48.367009 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:48.367366 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:48.367435 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:48.368904 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:48.368905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:48.368906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:48.368912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:48.370585 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:48.370996 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:48.370997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:48.370997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:48.371054 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:48.371275 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:48.371276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:48.371276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:48.371420 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:48.371520 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:48.371652 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 09:36:48.371752 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 09:36:48.373112 22332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec8c0efbd31a4cb58df8b8c71d911576: Generated new TSK 0
I20260926 09:36:48.373323 22332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:48.379370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22249
2026-09-26T09:36:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:48 dist-test-slave-21h8 krb5kdc[22348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:48 dist-test-slave-21h8 krb5kdc[22348](info): set up 2 sockets
Sep 26 09:36:48 dist-test-slave-21h8 krb5kdc[22348](info): commencing operation
krb5kdc: starting...
W20260926 09:36:49.419409 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:49 dist-test-slave-21h8 krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:49.531522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:49.610894 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:49.611074 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:49.611096 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:49.612161 22361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:49.612198 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:49.612218 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:49.612231 22361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:49.612242 22361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:49.613579 22361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:49.613744 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:49.613902 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:36:49.618592 22361 server_base.cc:1034] running on GCE node
W20260926 09:36:49.618700 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:49.618695 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:49.618695 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:49.619035 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:49.619354 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:49.620527 22361 hybrid_clock.cc:648] HybridClock initialized: now 1790415409620518 us; error 31 us; skew 500 ppm
Sep 26 09:36:49 dist-test-slave-21h8 krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:49.622888 22361 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:49.623381 22361 webserver.cc:460] Webserver started at http://127.0.98.62:32901/ using document root <none> and password file <none>
I20260926 09:36:49.623607 22361 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:49.623696 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:49.623858 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:49.625368 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "48c7c82164a64fbcb57efebdb717a7d6"
format_stamp: "Formatted at 2026-09-26 09:36:49 on dist-test-slave-21h8"
server_key: "1099b901e77003fa13d14b737968d25d"
server_key_iv: "fceb0105364b5fc88492e6b695cc7a21"
server_key_version: "encryptionkey@0"
I20260926 09:36:49.625787 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "48c7c82164a64fbcb57efebdb717a7d6"
format_stamp: "Formatted at 2026-09-26 09:36:49 on dist-test-slave-21h8"
server_key: "1099b901e77003fa13d14b737968d25d"
server_key_iv: "fceb0105364b5fc88492e6b695cc7a21"
server_key_version: "encryptionkey@0"
I20260926 09:36:49.627556 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260926 09:36:49.628486 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:49.628691 22361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:49.628798 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "48c7c82164a64fbcb57efebdb717a7d6"
format_stamp: "Formatted at 2026-09-26 09:36:49 on dist-test-slave-21h8"
server_key: "1099b901e77003fa13d14b737968d25d"
server_key_iv: "fceb0105364b5fc88492e6b695cc7a21"
server_key_version: "encryptionkey@0"
I20260926 09:36:49.628875 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:49.678547 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:49.680438 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:49.680547 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:49.684355 22361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35211
I20260926 09:36:49.684419 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35211 every 8 connection(s)
I20260926 09:36:49.684775 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:49.685652 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:49.685932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22361
I20260926 09:36:49.686053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:49.686236 392 external_mini_cluster.cc:1428] Setting key 3ab3932bcd5a29d039fb61595342f877
I20260926 09:36:49.688501 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6: Bootstrap starting.
I20260926 09:36:49.689195 22429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:49.689496 22429 log.cc:824] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:49.690181 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6: No bootstrap required, opened a new log
I20260926 09:36:49.691963 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:49.686778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52071 (local address 127.0.98.62:35211)
0926 09:36:49.686959 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:49.686974 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:49.687000 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:36:49.687467 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:49.687474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:49.687523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:49.687637 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:49.687913 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:49.688834 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:49.689951 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:49.690393 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:49.690485 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:49.691134 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:49.691138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:49.691138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:49.691295 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:49.691372 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:49.691387 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:49.691449 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:49.691555 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 09:36:49.691624 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260926 09:36:49.692091 22429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48c7c82164a64fbcb57efebdb717a7d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35211 } }
I20260926 09:36:49.692267 22429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:49.692303 22429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c7c82164a64fbcb57efebdb717a7d6, State: Initialized, Role: FOLLOWER
I20260926 09:36:49.692438 22429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48c7c82164a64fbcb57efebdb717a7d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35211 } }
I20260926 09:36:49.692521 22429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:49.692557 22429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:49.692605 22429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:49.693187 22429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48c7c82164a64fbcb57efebdb717a7d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35211 } }
I20260926 09:36:49.693290 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:49.693305 22429 leader_election.cc:302] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48c7c82164a64fbcb57efebdb717a7d6; no voters:
I20260926 09:36:49.693506 22429 leader_election.cc:288] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:49.693533 22434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:49.693646 22434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [term 1 LEADER]: Becoming Leader. State: Replica: 48c7c82164a64fbcb57efebdb717a7d6, State: Running, Role: LEADER
I20260926 09:36:49.693770 22429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:49.693779 22434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48c7c82164a64fbcb57efebdb717a7d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35211 } }
I20260926 09:36:49.694198 22436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48c7c82164a64fbcb57efebdb717a7d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48c7c82164a64fbcb57efebdb717a7d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35211 } } }
I20260926 09:36:49.694198 22435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48c7c82164a64fbcb57efebdb717a7d6. Latest consensus state: current_term: 1 leader_uuid: "48c7c82164a64fbcb57efebdb717a7d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48c7c82164a64fbcb57efebdb717a7d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35211 } } }
I20260926 09:36:49.694368 22436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:49.694463 22435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:49.694723 22447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:49.695199 22447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:49.698575 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:49.694997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57717 (local address 127.0.98.62:35211)
0926 09:36:49.695040 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:49.695042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:49.695269 (+ 227us) server_negotiation.cc:407] Connection header received
0926 09:36:49.695381 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:49.695382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:49.695395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:49.695472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:49.695687 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:49.696382 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:49.697333 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:49.697538 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:49.697684 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:49.698283 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:49.698284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:49.698285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:49.698289 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:49.698303 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:49.698307 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:49.698359 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:49.698440 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 09:36:49.698462 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:36:49.700271 22447 catalog_manager.cc:1347] Generated new cluster ID: b13a859dc3a4437bb7cf3316ec9804c5
I20260926 09:36:49.700326 22447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:49.709286 22447 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:49.709906 22447 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:49.723485 22447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48c7c82164a64fbcb57efebdb717a7d6: Generated new TSK 0
I20260926 09:36:49.723733 22447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:49.726694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22361
2026-09-26T09:36:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:49 dist-test-slave-21h8 krb5kdc[22460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:49 dist-test-slave-21h8 krb5kdc[22460](info): set up 2 sockets
Sep 26 09:36:49 dist-test-slave-21h8 krb5kdc[22460](info): commencing operation
krb5kdc: starting...
W20260926 09:36:50.763531 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:50 dist-test-slave-21h8 krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:50.878955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:50.959215 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:50.959484 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:50.959525 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:50.960676 22473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:50.960748 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:50.960772 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:50.960800 22473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:50.960831 22473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:50.962334 22473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:50.962620 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:50.962842 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:50.967558 22478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:50.967630 22473 server_base.cc:1034] running on GCE node
W20260926 09:36:50.967715 22479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:50.967563 22481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:50.968035 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:50.968366 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:50.969516 22473 hybrid_clock.cc:648] HybridClock initialized: now 1790415410969502 us; error 30 us; skew 500 ppm
Sep 26 09:36:50 dist-test-slave-21h8 krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:50.979900 22473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:50.980343 22473 webserver.cc:460] Webserver started at http://127.0.98.62:38283/ using document root <none> and password file <none>
I20260926 09:36:50.980577 22473 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:50.980631 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:50.980801 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:50.981842 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "3774e53cf36d4954869043e290b0ff4f"
format_stamp: "Formatted at 2026-09-26 09:36:50 on dist-test-slave-21h8"
server_key: "21940caf194c43240f4df9f4c8e92995"
server_key_iv: "7eda350e0d277f51c55cc05dac28312f"
server_key_version: "encryptionkey@0"
I20260926 09:36:50.982195 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "3774e53cf36d4954869043e290b0ff4f"
format_stamp: "Formatted at 2026-09-26 09:36:50 on dist-test-slave-21h8"
server_key: "21940caf194c43240f4df9f4c8e92995"
server_key_iv: "7eda350e0d277f51c55cc05dac28312f"
server_key_version: "encryptionkey@0"
I20260926 09:36:50.983613 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:36:50.984387 22488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:50.984597 22473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:50.984709 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "3774e53cf36d4954869043e290b0ff4f"
format_stamp: "Formatted at 2026-09-26 09:36:50 on dist-test-slave-21h8"
server_key: "21940caf194c43240f4df9f4c8e92995"
server_key_iv: "7eda350e0d277f51c55cc05dac28312f"
server_key_version: "encryptionkey@0"
I20260926 09:36:50.984788 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:50.988144 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:50.989893 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:50.990044 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:50.994000 22473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36647
I20260926 09:36:50.994031 22540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36647 every 8 connection(s)
I20260926 09:36:50.994341 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:50.995246 22541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:50.997393 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f: Bootstrap starting.
I20260926 09:36:50.997956 22541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:50.998229 22541 log.cc:824] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:50.998871 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f: No bootstrap required, opened a new log
I20260926 09:36:51.000464 22541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3774e53cf36d4954869043e290b0ff4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36647 } }
I20260926 09:36:51.000592 22541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:36:51.000613 22541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3774e53cf36d4954869043e290b0ff4f, State: Initialized, Role: FOLLOWER
I20260926 09:36:51.000782 22541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3774e53cf36d4954869043e290b0ff4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36647 } }
I20260926 09:36:51.000860 22541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:36:51.000909 22541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:36:51.000967 22541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:51.001475 22541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3774e53cf36d4954869043e290b0ff4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36647 } }
I20260926 09:36:51.001596 22541 leader_election.cc:302] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3774e53cf36d4954869043e290b0ff4f; no voters:
I20260926 09:36:51.001771 22541 leader_election.cc:288] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:36:51.001842 22544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:51.001987 22544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [term 1 LEADER]: Becoming Leader. State: Replica: 3774e53cf36d4954869043e290b0ff4f, State: Running, Role: LEADER
I20260926 09:36:51.002019 22541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:51.002075 22544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3774e53cf36d4954869043e290b0ff4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36647 } }
I20260926 09:36:51.002628 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3774e53cf36d4954869043e290b0ff4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3774e53cf36d4954869043e290b0ff4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36647 } } }
I20260926 09:36:51.002723 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:51.002640 22546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3774e53cf36d4954869043e290b0ff4f. Latest consensus state: current_term: 1 leader_uuid: "3774e53cf36d4954869043e290b0ff4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3774e53cf36d4954869043e290b0ff4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36647 } } }
I20260926 09:36:51.002954 22546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:51.002934 22553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:51.003623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22473
I20260926 09:36:51.003681 22553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:51.003743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:51.003934 392 external_mini_cluster.cc:1428] Setting key 0bbe26853366690e2567d3dee2c303bf
I20260926 09:36:51.005342 22553 catalog_manager.cc:1347] Generated new cluster ID: 20a250b34cbb4d50bd5b8e06f5a40f23
I20260926 09:36:51.005393 22553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:51.006225 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:51.004386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38231 (local address 127.0.98.62:36647)
0926 09:36:51.004536 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:51.004545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:51.004835 (+ 290us) server_negotiation.cc:407] Connection header received
0926 09:36:51.005000 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:51.005007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:51.005037 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:51.005225 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:51.005827 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:51.005830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:51.005830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:51.005835 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:51.005873 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:51.005887 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:51.005957 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:51.006019 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 09:36:51.006059 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260926 09:36:51.007056 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:51.009466 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:51.008443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35221 (local address 127.0.98.62:36647)
0926 09:36:51.008536 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:51.008538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:51.008581 (+ 43us) server_negotiation.cc:407] Connection header received
0926 09:36:51.008647 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:51.008648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:51.008665 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:51.008716 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:51.009256 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:51.009257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:51.009258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:36:51.009262 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:51.009275 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:36:51.009279 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:51.009339 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:51.009390 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 09:36:51.009410 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 09:36:51.028584 22553 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:51.028789 22553 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:51.034247 22553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3774e53cf36d4954869043e290b0ff4f: Generated new TSK 0
I20260926 09:36:51.034436 22553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:51.041700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22473
2026-09-26T09:36:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1314 ms)
[----------] 18 tests from AuthTokenIssuingTest (23646 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:51 dist-test-slave-21h8 krb5kdc[22572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:51 dist-test-slave-21h8 krb5kdc[22572](info): set up 2 sockets
Sep 26 09:36:51 dist-test-slave-21h8 krb5kdc[22572](info): commencing operation
krb5kdc: starting...
W20260926 09:36:52.071415 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:52.186168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39575,127.0.98.61:35735,127.0.98.60:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:52.265599 22585 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:36:52.265830 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:52.265859 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:52.265900 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:52.266980 22585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:52.267050 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:52.267093 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:52.267125 22585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:52.267149 22585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:52.268627 22585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39575,127.0.98.61:35735,127.0.98.60:35515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:52.268918 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:52.269177 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:52.273800 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:52.273819 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:52.273871 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:52.274085 22585 server_base.cc:1034] running on GCE node
I20260926 09:36:52.274307 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:52.274583 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:52.275743 22585 hybrid_clock.cc:648] HybridClock initialized: now 1790415412275730 us; error 29 us; skew 500 ppm
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:52.278116 22585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:52.278525 22585 webserver.cc:460] Webserver started at http://127.0.98.62:38421/ using document root <none> and password file <none>
I20260926 09:36:52.278748 22585 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:52.278815 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:52.278961 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:52.280092 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "c613cb7f695b4af994c709333f1dac44"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "a890b2e5ea04d76f2d368e260ed8073b"
server_key_iv: "d7d9e70573d8c6d0860cf8d9535ffdee"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.280449 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "c613cb7f695b4af994c709333f1dac44"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "a890b2e5ea04d76f2d368e260ed8073b"
server_key_iv: "d7d9e70573d8c6d0860cf8d9535ffdee"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.281744 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:36:52.282488 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:52.282712 22585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:36:52.282819 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "c613cb7f695b4af994c709333f1dac44"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "a890b2e5ea04d76f2d368e260ed8073b"
server_key_iv: "d7d9e70573d8c6d0860cf8d9535ffdee"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.282892 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:52.309320 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:52.311249 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:52.311424 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:52.315157 22585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39575
I20260926 09:36:52.315193 22652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39575 every 8 connection(s)
I20260926 09:36:52.315580 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:52.316550 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:52.317973 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:52.320477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22585
I20260926 09:36:52.320574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:52.320746 392 external_mini_cluster.cc:1428] Setting key 82ba98cfc02efd45071ca40c24f22d11
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:52.328115 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.318603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39575 (local address 127.0.0.1:43002)
0926 09:36:52.318758 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.318798 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.319245 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.319561 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.319563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.319786 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.319923 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.319926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.321244 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.321245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.321901 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.321903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.322437 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.322438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.322499 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.323005 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.323015 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.324573 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.326638 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.326644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.326645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.326840 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.327123 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.327124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.327124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.327222 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.327714 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.327720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.327776 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.327854 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.327969 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":108,"threads_started":1}
I20260926 09:36:52.328207 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.318650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.0.98.62:39575)
0926 09:36:52.318808 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.318811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.318875 (+ 64us) server_negotiation.cc:407] Connection header received
0926 09:36:52.319426 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.319428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.319444 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.319536 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.320004 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.321157 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.321988 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.322325 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.322451 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.324660 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.324669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.324669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.324676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.326543 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.326920 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.326923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.326923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.327022 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.327305 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.327306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.327306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.327615 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.327708 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.327912 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.328043 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260926 09:36:52.328928 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 09:36:52.329281 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.329163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:35980)
0926 09:36:52.329219 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.329265 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 09:36:52.329409 22601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
I20260926 09:36:52.329820 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.329729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:35990)
0926 09:36:52.329776 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.329799 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:36:52.329914 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35735: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 09:36:52.346506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35735
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39575,127.0.98.61:35735,127.0.98.60:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:36:52.379123 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } attempt: 1
I20260926 09:36:52.379670 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.379487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:35994)
0926 09:36:52.379595 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.379631 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 09:36:52.380038 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.379938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:36004)
0926 09:36:52.379985 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.380017 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:36:52.380131 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35735: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
W20260926 09:36:52.424845 22660 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:36:52.425087 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:52.425118 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:52.425161 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:52.426277 22660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:52.426344 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:52.426369 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:52.426391 22660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:52.426419 22660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:52.427891 22660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39575,127.0.98.61:35735,127.0.98.60:35515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:52.428154 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:52.428373 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:52.433166 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:52.433192 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:52.433318 22660 server_base.cc:1034] running on GCE node
W20260926 09:36:52.433409 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:52.433645 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:52.433951 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:52.435127 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790415412435113 us; error 27 us; skew 500 ppm
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:52.437780 22660 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 09:36:52.438266 22660 webserver.cc:460] Webserver started at http://127.0.98.61:38711/ using document root <none> and password file <none>
I20260926 09:36:52.438486 22660 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:52.438552 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:52.438700 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:52.439821 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/data/instance:
uuid: "f08e9abb26d14a5795def27e4ee7ce4e"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "a898cd73290d73b914dc9745b9b1534b"
server_key_iv: "d99db9c5c5f89a36aabddd385c19fe91"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.440173 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/wal/instance:
uuid: "f08e9abb26d14a5795def27e4ee7ce4e"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "a898cd73290d73b914dc9745b9b1534b"
server_key_iv: "d99db9c5c5f89a36aabddd385c19fe91"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.441505 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:52.442324 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:52.442502 22660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:52.442598 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/wal
uuid: "f08e9abb26d14a5795def27e4ee7ce4e"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "a898cd73290d73b914dc9745b9b1534b"
server_key_iv: "d99db9c5c5f89a36aabddd385c19fe91"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.442682 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:52.448297 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } attempt: 2
I20260926 09:36:52.448725 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.448560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:36014)
0926 09:36:52.448620 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.448666 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:36:52.449072 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.448974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:36018)
0926 09:36:52.449021 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.449049 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:36:52.449163 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35735: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35735: connect: Connection refused (error 111)
I20260926 09:36:52.472246 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:52.474140 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:52.474305 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:52.478401 22660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35735
I20260926 09:36:52.478463 22727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35735 every 8 connection(s)
I20260926 09:36:52.478804 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:36:52.479911 22728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:52.481069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22660
I20260926 09:36:52.481143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-1/wal/instance
I20260926 09:36:52.481333 392 external_mini_cluster.cc:1428] Setting key 82b2e759032759933ef6bd6f939b7961
I20260926 09:36:52.481760 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 09:36:52.491747 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.482497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.0.98.62:39575)
0926 09:36:52.482551 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.482553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.482822 (+ 269us) server_negotiation.cc:407] Connection header received
0926 09:36:52.483461 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.483463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.483480 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.483555 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.484210 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.485032 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.486215 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.486515 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.486592 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.488848 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.488849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.488850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.488857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.490181 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.490672 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.490674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.490675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.490738 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.491014 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.491015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.491016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.491139 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.491226 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.491610 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.491682 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:36:52.492044 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.482491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39575 (local address 127.0.0.1:43010)
0926 09:36:52.482670 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.482713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.483203 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.483630 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.483634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.483903 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.484103 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.484107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.485131 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.485132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.486091 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.486094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.486610 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.486612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.486862 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.487400 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.487407 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.488734 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.490270 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.490277 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.490279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.490540 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.490816 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.490817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.490817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.490920 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.491252 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.491258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.491443 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.491587 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.491760 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260926 09:36:52.492620 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:36:52.500299 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.492905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:36034)
0926 09:36:52.492953 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.492964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.493003 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.493264 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.493265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.493423 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.493481 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.493482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.494297 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.494298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.495201 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.495204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.495706 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.495707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.495791 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.496147 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.496153 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.497466 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.499185 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.499186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.499187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.499408 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.499622 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.499623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.499624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.499677 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.499994 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.499994 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.500032 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.500093 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.500190 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:36:52.500368 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.492965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.0.98.61:35735)
0926 09:36:52.493128 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.493132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.493148 (+ 16us) server_negotiation.cc:407] Connection header received
0926 09:36:52.493166 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.493167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.493177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.493240 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.493564 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.494216 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.495278 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.495647 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.495705 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.497565 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.497573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.497574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.497581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.499087 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.499498 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.499501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.499501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.499558 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.499756 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.499757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.499757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.499899 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.500003 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.500119 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.500241 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260926 09:36:52.500942 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:52.501272 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.501154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53338)
0926 09:36:52.501202 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.501247 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:36:52.501412 22679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
I20260926 09:36:52.501776 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.501688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53340)
0926 09:36:52.501731 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.501757 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:36:52.501879 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35515: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 09:36:52.506798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35515
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39575,127.0.98.61:35735,127.0.98.60:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:36:52.540300 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } attempt: 3
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:36:52.547462 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.540649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:36040)
0926 09:36:52.540704 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.540720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.540784 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.540951 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.540952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.541109 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.541172 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.541174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.541827 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.541828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.542354 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.542356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.542746 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.542747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.542830 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.543309 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.543349 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.544536 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.546377 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.546378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.546378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.546542 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.546750 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.546750 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.546750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.546799 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.547161 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.547162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.547199 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.547252 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.547371 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 09:36:52.547487 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.540720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.0.98.61:35735)
0926 09:36:52.540786 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.540789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.540799 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:52.540846 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.540847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.540859 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.540944 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.541253 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.541744 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.542450 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.542669 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.542748 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.544719 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.544721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.544721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.544729 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.546284 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.546611 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.546612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.546612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.546671 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.546937 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.546939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.546939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.547083 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.547167 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.547293 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.547397 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:36:52.548144 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:52.548483 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.548381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53348)
0926 09:36:52.548436 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.548465 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 09:36:52.548800 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.548710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53358)
0926 09:36:52.548753 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.548777 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:36:52.548895 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35515: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
I20260926 09:36:52.551091 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } attempt: 1
I20260926 09:36:52.551501 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.551381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53368)
0926 09:36:52.551439 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.551474 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 09:36:52.551838 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.551753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53376)
0926 09:36:52.551795 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.551818 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:36:52.551951 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35515: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
I20260926 09:36:52.580026 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } attempt: 1
I20260926 09:36:52.580420 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.580287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53388)
0926 09:36:52.580348 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.580395 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 09:36:52.580834 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.580741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53398)
0926 09:36:52.580794 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.580819 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 09:36:52.580957 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35515: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
W20260926 09:36:52.588148 22735 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:36:52.588335 22735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:52.588384 22735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:52.588418 22735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:52.589515 22735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:52.589552 22735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:52.589566 22735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:52.589617 22735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:52.589636 22735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:52.591051 22735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39575,127.0.98.61:35735,127.0.98.60:35515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:52.591281 22735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:52.591526 22735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:52.595988 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:52.595999 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:52.596143 22735 server_base.cc:1034] running on GCE node
W20260926 09:36:52.596213 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:52.596448 22735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:52.596763 22735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:52.597919 22735 hybrid_clock.cc:648] HybridClock initialized: now 1790415412597910 us; error 29 us; skew 500 ppm
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:52.600210 22735 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 09:36:52.600647 22735 webserver.cc:460] Webserver started at http://127.0.98.60:45023/ using document root <none> and password file <none>
I20260926 09:36:52.600857 22735 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:52.600898 22735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:52.601056 22735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:52.602058 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/data/instance:
uuid: "648c3cc272c844f79b9c2a1d63733cf5"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "7ab9c0c26100ee014dc50b89af710e72"
server_key_iv: "123306066d909b2183f42ac3dc6d0b42"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.602401 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/wal/instance:
uuid: "648c3cc272c844f79b9c2a1d63733cf5"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "7ab9c0c26100ee014dc50b89af710e72"
server_key_iv: "123306066d909b2183f42ac3dc6d0b42"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.603713 22735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:52.604414 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:52.604568 22735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:52.604662 22735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/wal
uuid: "648c3cc272c844f79b9c2a1d63733cf5"
format_stamp: "Formatted at 2026-09-26 09:36:52 on dist-test-slave-21h8"
server_key: "7ab9c0c26100ee014dc50b89af710e72"
server_key_iv: "123306066d909b2183f42ac3dc6d0b42"
server_key_version: "encryptionkey@0"
I20260926 09:36:52.604766 22735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:52.620083 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } attempt: 2
I20260926 09:36:52.620446 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.620327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53408)
0926 09:36:52.620396 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.620422 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:36:52.620765 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.620656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53412)
0926 09:36:52.620699 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.620721 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:36:52.620842 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35515: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35515: connect: Connection refused (error 111)
I20260926 09:36:52.623133 22735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:52.624962 22735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:52.625128 22735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:52.629223 22735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35515
I20260926 09:36:52.629252 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35515 every 8 connection(s)
I20260926 09:36:52.629613 22735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:36:52.630537 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:52.631006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22735
I20260926 09:36:52.631093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790415146110679-392-0/minicluster-data/master-2/wal/instance
I20260926 09:36:52.631372 392 external_mini_cluster.cc:1428] Setting key 5093eae84b2ac42b67ef21a3855b2458
I20260926 09:36:52.632593 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:36:52.641728 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.633307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.0.98.62:39575)
0926 09:36:52.633354 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.633356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.633468 (+ 112us) server_negotiation.cc:407] Connection header received
0926 09:36:52.633548 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.633549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.633559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.633618 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.634213 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.634991 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.635915 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.636118 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.636170 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.638340 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.638341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.638342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.638348 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.640152 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.640572 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.640573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.640573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.640632 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.640971 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.640971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.640972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.641111 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.641185 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.641517 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.641593 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:36:52.641968 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.633197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39575 (local address 127.0.0.1:43026)
0926 09:36:52.633363 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.633403 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.633479 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.633657 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.633659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.633955 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.634098 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.634104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.635056 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.635057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.635835 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.635836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.636213 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.636214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.636271 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.636872 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.636880 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.638253 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.640237 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.640243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.640244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.640488 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.640764 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.640765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.640766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.640899 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.641202 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.641209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.641368 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.641468 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.641626 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260926 09:36:52.642081 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.631918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.0.98.60:35515)
0926 09:36:52.632091 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.632100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.632132 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:36:52.632664 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.632672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.632732 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.632875 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.633570 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.634929 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.635759 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.636133 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.636253 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.638916 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.638923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.638924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.638930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.640697 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.641081 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.641084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.641084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.641137 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.641329 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.641329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.641329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.641445 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.641598 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.641712 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.641992 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260926 09:36:52.642803 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:52.649972 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.643040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35735 (local address 127.0.0.1:36048)
0926 09:36:52.643091 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.643102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.643149 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.643277 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.643277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.643452 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.643480 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.643481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.644110 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.644111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.644577 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.644579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.645063 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.645064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.645121 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.645704 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.645709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.646998 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.649083 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.649084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.649084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.649239 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.649456 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.649457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.649457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.649504 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.649777 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.649778 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.649807 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.649851 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.649917 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:36:52.650065 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.643137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.0.98.61:35735)
0926 09:36:52.643186 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.643188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.643196 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:36:52.643219 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.643220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.643228 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.643276 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.643560 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.644017 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.644661 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.644973 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.645054 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.647093 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.647094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.647094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.647101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.649002 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.649315 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.649317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.649317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.649384 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.649561 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.649562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.649563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.649706 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.649792 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.649881 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.649963 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:36:52.650385 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:52.650846 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.642850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.0.98.62:39575)
0926 09:36:52.642885 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.642887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.642894 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:36:52.643080 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.643080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.643088 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.643153 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.643671 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.644094 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.644780 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.645027 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.645101 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.647743 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.647744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.647745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.647753 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.649433 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.649848 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.649850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.649850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.649908 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.650125 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.650126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.650126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.650264 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.650423 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.650656 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.650739 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:36:52.656116 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } attempt: 2
I20260926 09:36:52.657696 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.650656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53426)
0926 09:36:52.650949 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.650960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.651010 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.651149 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.651149 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.651295 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.651357 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.651359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.652101 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.652102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.652590 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.652591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.652966 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.652967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.653013 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.653397 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.653402 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.654786 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.656610 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.656611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.656612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.656825 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.657034 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.657034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.657035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.657104 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.657393 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.657394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.657425 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.657490 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.657585 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260926 09:36:52.658011 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.650706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.0.98.60:35515)
0926 09:36:52.650746 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.650748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.651020 (+ 272us) server_negotiation.cc:407] Connection header received
0926 09:36:52.651050 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.651051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.651062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.651149 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.651434 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.652029 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.652669 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.652877 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.652966 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.654871 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.654872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.654873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.654879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.656535 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.656922 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.656923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.656924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.656973 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.657170 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.657171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.657171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.657302 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.657368 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.657860 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.657934 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:36:52.659152 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.651564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.0.98.61:35735)
0926 09:36:52.651607 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.651609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.651665 (+ 56us) server_negotiation.cc:407] Connection header received
0926 09:36:52.651730 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.651731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.651739 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.651794 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.652246 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.652686 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.653348 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.653552 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.653638 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.656169 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.656170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.656171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.656178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.657912 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.658342 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.658343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.658343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.658401 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.658630 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.658631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.658631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.658749 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.658896 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.658994 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.659072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:36:52.659571 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5: Bootstrap starting.
I20260926 09:36:52.660378 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:52.660691 22803 log.cc:824] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:52.660869 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:36:52.662312 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5: No bootstrap required, opened a new log
I20260926 09:36:52.662109 22627 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c613cb7f695b4af994c709333f1dac44"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43032
I20260926 09:36:52.664490 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.656346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53440)
0926 09:36:52.656658 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.656669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.656715 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.656886 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.656887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.657047 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.657113 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.657115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.658572 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.658573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.659358 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.659361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.659863 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.659864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.659924 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.660297 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.660302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.661525 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.663414 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.663415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.663415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.663585 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.663800 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.663801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.663802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.663858 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.664138 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.664138 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.664180 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.664263 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.664353 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294}
I20260926 09:36:52.664399 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } }
I20260926 09:36:52.664527 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.656401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.0.98.60:35515)
0926 09:36:52.656589 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.656591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.656722 (+ 131us) server_negotiation.cc:407] Connection header received
0926 09:36:52.656770 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.656771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.656783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.656871 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.657193 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.658471 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.659435 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.659781 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.659856 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.661615 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.661617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.661617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.661624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.663344 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.663672 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.663673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.663674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.663727 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.663932 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.663933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.663933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.664057 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.664131 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.664287 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.664355 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260926 09:36:52.664582 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648c3cc272c844f79b9c2a1d63733cf5, State: Initialized, Role: FOLLOWER
I20260926 09:36:52.664690 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } }
I20260926 09:36:52.664958 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.665023 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:52.665057 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:52.665709 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44: Bootstrap starting.
I20260926 09:36:52.666358 22653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:52.666667 22653 log.cc:824] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44: Log is configured to *not* fsync() on all Append() calls
W20260926 09:36:52.666981 22821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:36:52.667057 22821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:36:52.667265 22735 master_runner.cc:186] Error getting master registration for 127.0.98.62:39575: OK, instance_id { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" instance_seqno: 1790415412314121 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:36:52.669464 22735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:36:52.679981 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44: No bootstrap required, opened a new log
I20260926 09:36:52.680954 22653 raft_consensus.cc:348] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } }
I20260926 09:36:52.681087 22653 raft_consensus.cc:729] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c613cb7f695b4af994c709333f1dac44, State: Initialized, Role: FOLLOWER
I20260926 09:36:52.681174 22653 consensus_queue.cc:260] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } }
I20260926 09:36:52.681470 22653 sys_catalog.cc:564] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:52.681529 22627 raft_consensus.cc:482] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 09:36:52.681617 22627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:52.682355 22627 raft_consensus.cc:504] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } }
I20260926 09:36:52.682582 22627 leader_election.cc:288] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [CANDIDATE]: Term 1 election: Requested vote from peers f08e9abb26d14a5795def27e4ee7ce4e (127.0.98.61:35735), 648c3cc272c844f79b9c2a1d63733cf5 (127.0.98.60:35515)
W20260926 09:36:52.683265 22835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:36:52.683408 22835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:36:52.684006 22777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c613cb7f695b4af994c709333f1dac44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "648c3cc272c844f79b9c2a1d63733cf5"
I20260926 09:36:52.684123 22777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:52.684939 22777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c613cb7f695b4af994c709333f1dac44 in term 1.
I20260926 09:36:52.685199 22604 leader_election.cc:302] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648c3cc272c844f79b9c2a1d63733cf5, c613cb7f695b4af994c709333f1dac44; no voters:
I20260926 09:36:52.681488 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.685519 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:52.685458 22842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:52.685956 22842 raft_consensus.cc:686] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [term 1 LEADER]: Becoming Leader. State: Replica: c613cb7f695b4af994c709333f1dac44, State: Running, Role: LEADER
I20260926 09:36:52.686225 22842 consensus_queue.cc:237] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } }
I20260926 09:36:52.683956 22585 master_runner.cc:186] Error getting master registration for 127.0.98.61:35735: OK, instance_id { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" instance_seqno: 1790415412477241 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:36:52.686573 22585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:36:52.687160 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c613cb7f695b4af994c709333f1dac44. Latest consensus state: current_term: 1 leader_uuid: "c613cb7f695b4af994c709333f1dac44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.687297 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:52.687543 22845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:52.687727 22845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:52.689095 22777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [term 1 FOLLOWER]: Refusing update from remote peer c613cb7f695b4af994c709333f1dac44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:52.689417 22824 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:52.691134 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c613cb7f695b4af994c709333f1dac44. Latest consensus state: current_term: 1 leader_uuid: "c613cb7f695b4af994c709333f1dac44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.691222 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:52.692341 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c613cb7f695b4af994c709333f1dac44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.692428 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:52.692783 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c613cb7f695b4af994c709333f1dac44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.692865 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:52.693233 22848 mvcc.cc:205] Tried to move back new op lower bound from 7333541530372333568 to 7333541530363957248. Current Snapshot: MvccSnapshot[applied={T|T < 7333541530372333568}]
I20260926 09:36:52.693717 22849 mvcc.cc:205] Tried to move back new op lower bound from 7333541530372333568 to 7333541530363957248. Current Snapshot: MvccSnapshot[applied={T|T < 7333541530372333568}]
I20260926 09:36:52.695627 22845 catalog_manager.cc:1347] Generated new cluster ID: 0b0bcceb764f495c91aabe230f74e687
I20260926 09:36:52.695675 22845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:52.698925 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.684047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.0.98.61:35735)
0926 09:36:52.684094 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.684096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.688046 (+ 3950us) server_negotiation.cc:407] Connection header received
0926 09:36:52.688173 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.688174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.688195 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.688226 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.689485 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.690229 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.692032 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.692375 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.692417 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.695062 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.695063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.695064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.695081 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.696882 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.697440 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.697441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.697441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.697506 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.697793 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.697794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.697794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.697927 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.698423 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.698755 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.698832 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:52.701817 22845 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:52.702486 22845 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:52.709225 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.687719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.0.98.60:35515)
0926 09:36:52.687759 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.687761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.699489 (+ 11728us) server_negotiation.cc:407] Connection header received
0926 09:36:52.699626 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.699627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.699638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.699676 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.700487 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.701257 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.702849 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.703167 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.703207 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.705318 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.705319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.705320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.705345 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.707214 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.707929 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.707930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.707930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.707986 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.708384 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.708385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.708386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.708538 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.708655 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.709022 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.709116 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:52.709383 22845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44: Generated new TSK 0
I20260926 09:36:52.709543 22845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:36:52.710608 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.687529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.0.98.62:39575)
0926 09:36:52.687572 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.687574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.692293 (+ 4719us) server_negotiation.cc:407] Connection header received
0926 09:36:52.692562 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.692563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.692583 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.692611 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.703695 (+ 11084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.704418 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.705422 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.705720 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.705762 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.707423 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.707424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.707424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.707431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.709128 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.709525 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.709526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.709526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.709582 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.709788 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.709789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.709789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.709922 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.710031 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.710417 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.710508 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:36:52.711987 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } attempt: 3
Sep 26 09:36:52 dist-test-slave-21h8 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:36:52.721222 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.712215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35515 (local address 127.0.0.1:53450)
0926 09:36:52.712261 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:52.712272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:52.712308 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:52.712452 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:52.712453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:52.712608 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:52.712649 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.712650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.714710 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.714711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.715545 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:52.715548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.716365 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.716366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:52.716430 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.716791 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:52.716796 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:52.718260 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:52.720037 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.720038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.720038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.720271 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.720524 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:52.720524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:52.720525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:52.720591 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:52.720865 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:52.720866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:52.720931 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 09:36:52.720997 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:36:52.721097 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:36:52.721531 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:52.712279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.0.98.60:35515)
0926 09:36:52.712309 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.712311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.712319 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:36:52.712360 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.712361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.712370 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.712414 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.713885 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.714613 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.715912 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:52.716252 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:52.716316 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:52.718350 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:52.718351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:52.718351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:52.718358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:52.719955 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.720360 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.720361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.720361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.720425 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:52.720653 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:52.720653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:52.720654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:52.720769 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:52.720824 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:52.721373 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 09:36:52.721448 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:36:52.723129 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e: Bootstrap starting.
I20260926 09:36:52.723922 22728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:52.724236 22728 log.cc:824] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:52.725147 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e: No bootstrap required, opened a new log
I20260926 09:36:52.727137 22728 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } }
I20260926 09:36:52.727268 22728 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08e9abb26d14a5795def27e4ee7ce4e, State: Initialized, Role: FOLLOWER
I20260926 09:36:52.727389 22728 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } }
I20260926 09:36:52.727715 22728 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:52.727799 22702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c613cb7f695b4af994c709333f1dac44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f08e9abb26d14a5795def27e4ee7ce4e"
I20260926 09:36:52.727916 22702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:52.728694 22702 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c613cb7f695b4af994c709333f1dac44 in term 1.
I20260926 09:36:52.729059 22701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [term 1 FOLLOWER]: Refusing update from remote peer c613cb7f695b4af994c709333f1dac44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:52.729375 22843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:36:52.729445 22871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:36:52.729482 22871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:36:52.729894 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.729980 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:52.735405 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c613cb7f695b4af994c709333f1dac44. Latest consensus state: current_term: 1 leader_uuid: "c613cb7f695b4af994c709333f1dac44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.735482 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:52.735517 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c613cb7f695b4af994c709333f1dac44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:52.735559 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:52.739861 22874 mvcc.cc:205] Tried to move back new op lower bound from 7333541530452602880 to 7333541530363957248. Current Snapshot: MvccSnapshot[applied={T|T < 7333541530372333568}]
W20260926 09:36:52.740576 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.736370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.98.60:35515)
0926 09:36:52.736404 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.736405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.740058 (+ 3653us) server_negotiation.cc:407] Connection header received
0926 09:36:52.740153 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.740154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.740162 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.740202 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.740510 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53454: BlockingRecv error: recv got EOF from 127.0.0.1:53454 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 09:36:52.740751 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.735902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.0.98.61:35735)
0926 09:36:52.735940 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.735942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.739544 (+ 3602us) server_negotiation.cc:407] Connection header received
0926 09:36:52.739790 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.739791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.739800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.739828 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.740521 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36060: BlockingRecv error: recv got EOF from 127.0.0.1:36060 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:36:52.741065 22656 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.736146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.0.98.62:39575)
0926 09:36:52.736200 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.736202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.739643 (+ 3441us) server_negotiation.cc:407] Connection header received
0926 09:36:52.739707 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.739708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.739721 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.739783 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.741013 (+ 1230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43050: BlockingRecv error: recv got EOF from 127.0.0.1:43050 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 09:36:52.743584 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.742853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.0.98.61:35735)
0926 09:36:52.742897 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.742899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.743060 (+ 161us) server_negotiation.cc:407] Connection header received
0926 09:36:52.743156 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.743157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.743172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.743199 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.743548 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36076: BlockingRecv error: recv got EOF from 127.0.0.1:36076 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:36:52.746661 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.745941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.98.61:35735)
0926 09:36:52.745972 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.745973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.746209 (+ 236us) server_negotiation.cc:407] Connection header received
0926 09:36:52.746298 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.746299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.746307 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.746334 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.746621 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36092: BlockingRecv error: recv got EOF from 127.0.0.1:36092 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:36:52.750481 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.749791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.0.98.61:35735)
0926 09:36:52.749822 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.749823 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.750020 (+ 197us) server_negotiation.cc:407] Connection header received
0926 09:36:52.750108 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.750109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.750118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.750145 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.750443 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36102: BlockingRecv error: recv got EOF from 127.0.0.1:36102 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 09:36:52.756443 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.755751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.0.98.61:35735)
0926 09:36:52.755783 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.755784 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.755983 (+ 199us) server_negotiation.cc:407] Connection header received
0926 09:36:52.756072 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.756073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.756082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.756109 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.756403 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36106: BlockingRecv error: recv got EOF from 127.0.0.1:36106 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 09:36:52.766297 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.765536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.0.98.61:35735)
0926 09:36:52.765567 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.765569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.765770 (+ 201us) server_negotiation.cc:407] Connection header received
0926 09:36:52.765923 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.765923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.765933 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.765962 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.766247 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36122: BlockingRecv error: recv got EOF from 127.0.0.1:36122 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 09:36:52.784507 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.783671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.0.98.61:35735)
0926 09:36:52.783707 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.783709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.783947 (+ 238us) server_negotiation.cc:407] Connection header received
0926 09:36:52.784044 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.784045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.784056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.784085 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.784455 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36132: BlockingRecv error: recv got EOF from 127.0.0.1:36132 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 09:36:52.818903 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.817973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.0.98.61:35735)
0926 09:36:52.818019 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.818022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.818284 (+ 262us) server_negotiation.cc:407] Connection header received
0926 09:36:52.818395 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.818397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.818409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.818441 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.818834 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36134: BlockingRecv error: recv got EOF from 127.0.0.1:36134 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 09:36:52.885152 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:52.884510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.0.98.61:35735)
0926 09:36:52.884561 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:52.884563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:52.884650 (+ 87us) server_negotiation.cc:407] Connection header received
0926 09:36:52.884688 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:52.884689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:52.884701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:52.884782 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:52.885089 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36144: BlockingRecv error: recv got EOF from 127.0.0.1:36144 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 09:36:53.015017 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:53.014344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.0.98.61:35735)
0926 09:36:53.014431 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:53.014433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:53.014457 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:36:53.014541 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:53.014542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:53.014552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:53.014605 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:53.014956 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36146: BlockingRecv error: recv got EOF from 127.0.0.1:36146 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 09:36:53.251626 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c613cb7f695b4af994c709333f1dac44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613cb7f695b4af994c709333f1dac44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } peers { permanent_uuid: "f08e9abb26d14a5795def27e4ee7ce4e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35735 } } peers { permanent_uuid: "648c3cc272c844f79b9c2a1d63733cf5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35515 } } }
I20260926 09:36:53.251783 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P c613cb7f695b4af994c709333f1dac44 [sys.catalog]: This master's current role is: LEADER
W20260926 09:36:53.272671 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:53.272103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.0.98.61:35735)
0926 09:36:53.272169 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:53.272171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:53.272199 (+ 28us) server_negotiation.cc:407] Connection header received
0926 09:36:53.272258 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:53.272259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:53.272268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:53.272335 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:53.272625 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36156: BlockingRecv error: recv got EOF from 127.0.0.1:36156 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:36:53.667541 22821 catalog_manager.cc:1259] Loaded cluster ID: 0b0bcceb764f495c91aabe230f74e687
I20260926 09:36:53.667609 22821 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5: loading cluster ID for follower catalog manager: success
I20260926 09:36:53.668416 22821 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5: acquiring CA information for follower catalog manager: success
I20260926 09:36:53.668738 22821 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 648c3cc272c844f79b9c2a1d63733cf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:36:53.729946 22871 catalog_manager.cc:1259] Loaded cluster ID: 0b0bcceb764f495c91aabe230f74e687
I20260926 09:36:53.730026 22871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e: loading cluster ID for follower catalog manager: success
I20260926 09:36:53.730911 22871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e: acquiring CA information for follower catalog manager: success
I20260926 09:36:53.731256 22871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f08e9abb26d14a5795def27e4ee7ce4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:36:53.788996 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:53.786080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.0.98.61:35735)
0926 09:36:53.786213 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:53.786215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:53.786319 (+ 104us) server_negotiation.cc:407] Connection header received
0926 09:36:53.786420 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:53.786421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:53.786499 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:53.786594 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:53.786755 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.787269 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.788118 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.788340 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.788421 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:53.788601 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:53.788733 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:53.788831 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:53.788874 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 09:36:53.788897 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
I20260926 09:36:53.793179 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:53.790888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.0.98.60:35515)
0926 09:36:53.791002 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:53.791003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:53.791011 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:36:53.791047 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:53.791047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:53.791080 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:53.791161 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:53.791231 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.791768 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.792459 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.792671 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.792731 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:53.792838 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:53.792931 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:53.792985 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:53.793026 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 09:36:53.793042 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260926 09:36:53.797787 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:53.795026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.98.60:35515)
0926 09:36:53.795066 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:53.795068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:53.795179 (+ 111us) server_negotiation.cc:407] Connection header received
0926 09:36:53.795276 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:53.795277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:53.795286 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:53.795341 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:53.795533 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.796000 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.797047 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.797282 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.797312 (+ 30us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:53.797529 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:53.797559 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:53.797586 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:53.797679 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 09:36:53.797706 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:36:53.797925 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:53.795038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.0.98.61:35735)
0926 09:36:53.795074 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:53.795076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:53.795085 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:36:53.795153 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:53.795154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:53.795165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:53.795225 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:53.795402 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.796151 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.797118 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.797409 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.797471 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:53.797643 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:53.797674 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:53.797716 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:53.797820 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 09:36:53.797848 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:53.798024 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:53.794898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.0.98.62:39575)
0926 09:36:53.795105 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:53.795107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:53.795223 (+ 116us) server_negotiation.cc:407] Connection header received
0926 09:36:53.795303 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:53.795304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:53.795372 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:53.795453 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:53.795563 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.796290 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.797033 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:53.797324 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:53.797389 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:53.797584 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:53.797708 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:53.797763 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:53.797919 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:36:53.797941 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":50,"threads_started":1}
I20260926 09:36:53.799943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22585
I20260926 09:36:53.806587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22660
I20260926 09:36:53.811288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22735
2026-09-26T09:36:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2772 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:36:53 dist-test-slave-21h8 krb5kdc[22957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:36:53 dist-test-slave-21h8 krb5kdc[22957](info): set up 2 sockets
Sep 26 09:36:53 dist-test-slave-21h8 krb5kdc[22957](info): commencing operation
krb5kdc: starting...
W20260926 09:36:54.851750 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/test-user.keytab.
Sep 26 09:36:54 dist-test-slave-21h8 krb5kdc[22957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:36:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:36:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:36:54.966207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45205,127.0.98.61:36921,127.0.98.60:40845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:36:55.045252 22970 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:36:55.045499 22970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:55.045529 22970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:55.045575 22970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:55.046684 22970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:55.046748 22970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:55.046772 22970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:55.046794 22970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:55.046821 22970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:55.048301 22970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45205,127.0.98.61:36921,127.0.98.60:40845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:55.048579 22970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:55.048796 22970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:55.053697 22975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:55.053730 22978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:55.053967 22970 server_base.cc:1034] running on GCE node
W20260926 09:36:55.053745 22976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:55.054270 22970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:55.054563 22970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:55.055733 22970 hybrid_clock.cc:648] HybridClock initialized: now 1790415415055711 us; error 33 us; skew 500 ppm
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:55.058202 22970 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:36:55.058615 22970 webserver.cc:460] Webserver started at http://127.0.98.62:36083/ using document root <none> and password file <none>
I20260926 09:36:55.058848 22970 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:55.058916 22970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:55.059064 22970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:55.060204 22970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/data/instance:
uuid: "af1857b7f696428dad9fb61f73355abc"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "3f0a7d3d448a4f41eaadab60efeb235b"
server_key_iv: "e0a06448d69b594b83c14f211460fbcf"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.060567 22970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/wal/instance:
uuid: "af1857b7f696428dad9fb61f73355abc"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "3f0a7d3d448a4f41eaadab60efeb235b"
server_key_iv: "e0a06448d69b594b83c14f211460fbcf"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.061842 22970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:36:55.062561 22985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:55.062724 22970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:36:55.062820 22970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/wal
uuid: "af1857b7f696428dad9fb61f73355abc"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "3f0a7d3d448a4f41eaadab60efeb235b"
server_key_iv: "e0a06448d69b594b83c14f211460fbcf"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.062902 22970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:55.090045 22970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:55.092406 22970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:55.092516 22970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:55.096318 22970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45205
I20260926 09:36:55.096378 23037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45205 every 8 connection(s)
I20260926 09:36:55.096715 22970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:36:55.097693 23038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:55.099231 23038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:55.100387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 22970
I20260926 09:36:55.100461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-0/wal/instance
I20260926 09:36:55.100654 392 external_mini_cluster.cc:1428] Setting key 152057176ea0656bc087814ac5c10971
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 09:36:55.109684 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.099995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.0.98.62:45205)
0926 09:36:55.100162 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.100171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.100293 (+ 122us) server_negotiation.cc:407] Connection header received
0926 09:36:55.100867 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.100870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.100893 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.100999 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.101501 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.102534 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.103615 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.104140 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.104277 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.106229 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.106237 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.106238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.106245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.108092 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.108488 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.108492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.108492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.108573 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.108805 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.108806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.108806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.109015 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.109109 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.109315 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.109448 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260926 09:36:55.109691 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.099980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45205 (local address 127.0.0.1:34852)
0926 09:36:55.100149 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.100187 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.100678 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.101015 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.101017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.101246 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.101405 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.101408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.102632 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.102633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.103537 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.103539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.104261 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.104262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.104315 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.104839 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.104847 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.106142 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.108172 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.108178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.108180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.108375 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.108653 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.108654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.108654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.108733 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.109088 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.109092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.109154 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.109276 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.109511 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":100,"thread_start_us":97,"threads_started":1}
I20260926 09:36:55.110558 23038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260926 09:36:55.110930 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.110790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:40634)
0926 09:36:55.110839 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.110882 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:36:55.111040 22987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
I20260926 09:36:55.111428 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.111357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:40640)
0926 09:36:55.111384 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.111407 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 09:36:55.111532 23038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 09:36:55.125811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36921
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45205,127.0.98.61:36921,127.0.98.60:40845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:36:55.160758 23038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } attempt: 1
I20260926 09:36:55.161206 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.161079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:40652)
0926 09:36:55.161143 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.161178 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 09:36:55.161635 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.161555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:40656)
0926 09:36:55.161596 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.161619 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:36:55.161751 23038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
W20260926 09:36:55.205801 23045 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:36:55.206002 23045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:55.206019 23045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:55.206066 23045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:55.207201 23045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:55.207242 23045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:55.207255 23045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:55.207296 23045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:55.207338 23045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:55.208832 23045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45205,127.0.98.61:36921,127.0.98.60:40845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:55.209069 23045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:55.209290 23045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:55.214138 23051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:55.214177 23050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:55.214176 23045 server_base.cc:1034] running on GCE node
W20260926 09:36:55.214147 23053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:55.214547 23045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:55.214880 23045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:55.216058 23045 hybrid_clock.cc:648] HybridClock initialized: now 1790415415216044 us; error 43 us; skew 500 ppm
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:55.218478 23045 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 09:36:55.218950 23045 webserver.cc:460] Webserver started at http://127.0.98.61:42759/ using document root <none> and password file <none>
I20260926 09:36:55.219177 23045 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:55.219244 23045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:55.219414 23045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:55.220494 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/data/instance:
uuid: "69803b893503470f93b2e825c72d2f40"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "2153503d5b30d0e5eb73e5b53dc90914"
server_key_iv: "161b2bf4805ec1bef0142987390fd028"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.220856 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/wal/instance:
uuid: "69803b893503470f93b2e825c72d2f40"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "2153503d5b30d0e5eb73e5b53dc90914"
server_key_iv: "161b2bf4805ec1bef0142987390fd028"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.222122 23045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:36:55.222810 23060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:55.222981 23045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:55.223080 23045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/wal
uuid: "69803b893503470f93b2e825c72d2f40"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "2153503d5b30d0e5eb73e5b53dc90914"
server_key_iv: "161b2bf4805ec1bef0142987390fd028"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.223156 23045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:55.229928 23038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } attempt: 2
I20260926 09:36:55.230304 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.230185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:40668)
0926 09:36:55.230243 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.230278 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:36:55.230674 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.230578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:40674)
0926 09:36:55.230623 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.230649 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:36:55.230752 23038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36921: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36921: connect: Connection refused (error 111)
I20260926 09:36:55.231455 23045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:55.233153 23045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:55.233304 23045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:55.237251 23045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36921
I20260926 09:36:55.237279 23112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36921 every 8 connection(s)
I20260926 09:36:55.237615 23045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:36:55.238551 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:55.240136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 23045
I20260926 09:36:55.240216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-1/wal/instance
I20260926 09:36:55.240175 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:55.240414 392 external_mini_cluster.cc:1428] Setting key 0b797a17711afacfc159cf9f17e3233e
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 09:36:55.249730 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.240916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.0.98.62:45205)
0926 09:36:55.240986 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.240988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.241227 (+ 239us) server_negotiation.cc:407] Connection header received
0926 09:36:55.241955 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.241957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.241969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.242049 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.242625 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.243109 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.244146 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.244444 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.244529 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.246867 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.246869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.246869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.246876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.248318 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.248744 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.248745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.248746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.248806 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.249076 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.249077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.249077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.249193 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.249264 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.249553 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.249639 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:36:55.249886 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.240896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45205 (local address 127.0.0.1:34864)
0926 09:36:55.241087 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.241135 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.241851 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.242100 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.242103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.242339 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.242535 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.242540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.243194 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.243195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.244047 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.244050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.244561 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.244563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.244678 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.245191 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.245202 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.246758 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.248421 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.248428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.248430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.248682 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.248891 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.248892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.248893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.249001 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.249267 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.249275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.249415 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.249520 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.249666 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260926 09:36:55.250451 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:36:55.258562 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.250726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:40680)
0926 09:36:55.250776 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.250788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.250829 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.251143 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.251144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.251302 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.251411 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.251413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.252660 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.252661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.253493 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.253496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.253946 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.253947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.254031 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.254374 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.254379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.255704 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.257454 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.257455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.257456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.257618 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.257857 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.257858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.257858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.257912 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.258201 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.258202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.258239 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.258309 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.258403 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:36:55.258620 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.250830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.98.61:36921)
0926 09:36:55.250974 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.250979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.251002 (+ 23us) server_negotiation.cc:407] Connection header received
0926 09:36:55.251037 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.251039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.251052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.251132 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.251485 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.252571 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.253573 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.253868 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.253923 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.255800 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.255808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.255808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.255814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.257351 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.257732 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.257735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.257735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.257790 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.257981 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.257982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.257982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.258115 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.258219 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.258338 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.258454 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260926 09:36:55.259254 23113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:55.259605 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.259499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:43242)
0926 09:36:55.259544 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.259587 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:36:55.259737 23062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
I20260926 09:36:55.260165 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.260080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:43250)
0926 09:36:55.260128 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.260151 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 09:36:55.260267 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40845: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 09:36:55.264246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TYWI0/build/release/bin/kudu
/tmp/dist-test-task5TYWI0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40845
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45205,127.0.98.61:36921,127.0.98.60:40845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:36:55.309479 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } attempt: 1
I20260926 09:36:55.309978 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.309839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:43258)
0926 09:36:55.309908 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.309948 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 09:36:55.310341 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.310245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:43270)
0926 09:36:55.310292 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.310316 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:36:55.310441 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40845: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
I20260926 09:36:55.321915 23038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } attempt: 3
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:36:55.329057 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.322236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:40684)
0926 09:36:55.322292 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.322304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.322385 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.322548 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.322549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.322718 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.322782 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.322785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.323512 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.323514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.324033 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.324034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.324431 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.324432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.324514 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.324973 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.324985 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.326242 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.327986 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.327987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.327988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.328155 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.328386 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.328387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.328388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.328457 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.328722 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.328723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.328770 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.328855 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.328955 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:36:55.329061 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.322308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.0.98.61:36921)
0926 09:36:55.322372 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.322375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.322385 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:55.322458 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.322459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.322471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.322541 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.322869 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.323424 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.324132 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.324357 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.324433 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.326338 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.326340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.326340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.326347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.327908 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.328245 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.328246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.328247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.328304 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.328522 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.328523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.328523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.328636 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.328728 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.328892 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.328978 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:36:55.329440 23038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:55.329777 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.329680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:43274)
0926 09:36:55.329728 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.329753 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:36:55.330111 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.330024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:43284)
0926 09:36:55.330069 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.330091 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:36:55.330235 23038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40845: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
W20260926 09:36:55.348850 23120 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:36:55.349066 23120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:36:55.349083 23120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:36:55.349131 23120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:36:55.350271 23120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:36:55.350311 23120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:36:55.350325 23120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:36:55.350385 23120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:36:55.350414 23120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:36:55.352002 23120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45205,127.0.98.61:36921,127.0.98.60:40845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:36:55.352252 23120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:36:55.352478 23120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:36:55.357241 23126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:55.357288 23120 server_base.cc:1034] running on GCE node
W20260926 09:36:55.357249 23125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:36:55.357259 23128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:36:55.357589 23120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:36:55.358023 23120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:36:55.359181 23120 hybrid_clock.cc:648] HybridClock initialized: now 1790415415359170 us; error 42 us; skew 500 ppm
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:36:55.361368 23038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } attempt: 1
I20260926 09:36:55.361737 23120 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 09:36:55.361769 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.361622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:43300)
0926 09:36:55.361702 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.361732 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 09:36:55.362146 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.362033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:43308)
0926 09:36:55.362099 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.362123 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 09:36:55.362174 23120 webserver.cc:460] Webserver started at http://127.0.98.60:39513/ using document root <none> and password file <none>
W20260926 09:36:55.362229 23038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40845: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
I20260926 09:36:55.362424 23120 fs_manager.cc:359] Metadata directory not provided
I20260926 09:36:55.362471 23120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:36:55.362612 23120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:36:55.363696 23120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/data/instance:
uuid: "6fe16528a4b948728dda65acfcea87cc"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "097e7b5905384ffa7e6e34408346679e"
server_key_iv: "5c8d68d513bcaead030fd3bc13d7aa86"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.364053 23120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/wal/instance:
uuid: "6fe16528a4b948728dda65acfcea87cc"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "097e7b5905384ffa7e6e34408346679e"
server_key_iv: "5c8d68d513bcaead030fd3bc13d7aa86"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.365376 23120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:36:55.366096 23135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:36:55.366264 23120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:36:55.366358 23120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/wal
uuid: "6fe16528a4b948728dda65acfcea87cc"
format_stamp: "Formatted at 2026-09-26 09:36:55 on dist-test-slave-21h8"
server_key: "097e7b5905384ffa7e6e34408346679e"
server_key_iv: "5c8d68d513bcaead030fd3bc13d7aa86"
server_key_version: "encryptionkey@0"
I20260926 09:36:55.366438 23120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:36:55.378142 23120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:36:55.378580 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } attempt: 2
I20260926 09:36:55.378975 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.378832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:42864)
0926 09:36:55.378898 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.378939 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 09:36:55.379379 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.379249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:42872)
0926 09:36:55.379299 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.379345 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 09:36:55.379496 23113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40845: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40845: connect: Connection refused (error 111)
I20260926 09:36:55.380275 23120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:36:55.380429 23120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:36:55.384087 23120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40845
I20260926 09:36:55.384132 23187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40845 every 8 connection(s)
I20260926 09:36:55.384435 23120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:36:55.385421 23188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:36:55.386826 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:55.388470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TYWI0/build/release/bin/kudu as pid 23120
I20260926 09:36:55.388597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TYWI0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790415146110679-392-0/minicluster-data/master-2/wal/instance
I20260926 09:36:55.388787 392 external_mini_cluster.cc:1428] Setting key 235451732f1265d054441e6aa96c4db4
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:36:55.396226 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.387427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.0.98.62:45205)
0926 09:36:55.387498 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.387500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.387723 (+ 223us) server_negotiation.cc:407] Connection header received
0926 09:36:55.388128 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.388129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.388138 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.388188 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.389018 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.389711 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.390703 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.390911 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.390987 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.393017 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.393018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.393019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.393026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.394599 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.395041 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.395043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.395043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.395122 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.395424 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.395426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.395427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.395580 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.395670 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.396036 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.396122 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:36:55.396358 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.387471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45205 (local address 127.0.0.1:48706)
0926 09:36:55.387609 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.387641 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.388053 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.388237 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.388240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.388621 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.388925 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.388931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.389804 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.389804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.390594 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.390596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.391013 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.391015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.391137 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.391628 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.391635 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.392926 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.394732 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.394739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.394741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.394958 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.395196 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.395197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.395198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.395356 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.395673 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.395679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.395842 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.395960 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.396120 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260926 09:36:55.396900 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:55.398167 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.389225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.0.98.60:40845)
0926 09:36:55.389417 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.389421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.389443 (+ 22us) server_negotiation.cc:407] Connection header received
0926 09:36:55.389464 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.389466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.389483 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.389561 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.390213 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.390996 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.391970 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.392280 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.392374 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.395112 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.395115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.395115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.395119 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.397027 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.397439 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.397441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.397442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.397502 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.397707 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.397708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.397708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.397840 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.397950 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.398052 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.398107 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":112,"threads_started":1}
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:36:55.404345 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.397165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36921 (local address 127.0.0.1:50912)
0926 09:36:55.397274 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.397283 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.397312 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.397432 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.397433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.397581 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.397612 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.397613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.398408 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.398409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.399143 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.399145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.399544 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.399545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.399613 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.399964 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.399967 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.401106 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.403306 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.403307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.403307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.403509 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.403730 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.403730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.403731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.403788 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.404048 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.404049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.404088 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.404167 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.404259 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 09:36:55.404367 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.397227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.0.98.61:36921)
0926 09:36:55.397298 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.397300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.397310 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:55.397355 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.397356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.397367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.397424 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.397691 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.398341 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.399215 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.399476 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.399525 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.401182 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.401183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.401184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.401189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.402720 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.403599 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.403600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.403600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.403663 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.403859 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.403860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.403860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.403976 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.404045 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.404208 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.404279 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:36:55.404706 23188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:36:55.406497 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.399027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.0.98.62:45205)
0926 09:36:55.399059 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.399060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.399069 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:36:55.399095 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.399095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.399101 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.399155 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.399755 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.400280 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.401005 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.401282 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.401328 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.403675 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.403676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.403677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.403683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.405506 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.405808 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.405809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.405809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.405873 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.406043 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.406043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.406044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.406168 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.406263 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.406387 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.406442 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:36:55.411535 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.405000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.0.98.60:40845)
0926 09:36:55.405032 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.405034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.405045 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:36:55.405070 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.405071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.405080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.405145 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.405338 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.405992 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.406890 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.407207 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.407269 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.408936 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.408937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.408938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.408944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.410296 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.410793 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.410794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.410794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.410847 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.411023 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.411024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.411024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.411124 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.411195 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.411339 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.411450 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":16}
I20260926 09:36:55.411741 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.404933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:42882)
0926 09:36:55.404982 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.404992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.405025 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.405147 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.405147 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.405247 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.405271 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.405272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.406086 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.406089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.406830 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.406832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.407277 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.407278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.407354 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.407742 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.407747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.408841 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.410557 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.410557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.410558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.410702 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.410903 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.410904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.410904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.410954 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.411189 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.411190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.411226 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.411292 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.411404 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:36:55.413154 23188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc: Bootstrap starting.
I20260926 09:36:55.413832 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.407124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.0.98.61:36921)
0926 09:36:55.407157 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.407159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.407165 (+ 6us) server_negotiation.cc:407] Connection header received
0926 09:36:55.407196 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.407197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.407205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.407277 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.407748 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.408262 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.408974 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.409183 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.409257 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.411216 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.411217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.411217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.411224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.412717 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.413049 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.413051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.413051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.413105 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.413308 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.413309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.413310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.413439 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.413598 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.413642 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.413732 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:36:55.414165 23188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:55.414412 23188 log.cc:824] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:55.415228 23188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc: No bootstrap required, opened a new log
I20260926 09:36:55.415340 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:36:55.416865 23188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } }
I20260926 09:36:55.416380 23012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "af1857b7f696428dad9fb61f73355abc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48718
I20260926 09:36:55.417002 23188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe16528a4b948728dda65acfcea87cc, State: Initialized, Role: FOLLOWER
I20260926 09:36:55.417092 23188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } }
I20260926 09:36:55.417338 23194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.417405 23188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:55.417423 23194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:36:55.419104 23205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:36:55.419170 23205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:36:55.419395 23120 master_runner.cc:186] Error getting master registration for 127.0.98.62:45205: OK, instance_id { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" instance_seqno: 1790415415095228 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:36:55.420922 23120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:36:55.437383 23038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } attempt: 2
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:36:55.445328 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.437724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:42888)
0926 09:36:55.437780 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.437792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.437840 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.438021 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.438022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.438229 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.438282 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.438284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.439181 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.439182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.440035 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.440037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.440504 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.440505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.440567 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.441129 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.441143 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.442612 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.444266 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.444267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.444268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.444443 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.444657 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.444657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.444658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.444716 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.445005 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.445005 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.445048 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.445136 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.445231 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:36:55.445328 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.437819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.0.98.60:40845)
0926 09:36:55.437876 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.437879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.437889 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:55.437915 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.437916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.437926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.437999 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.438372 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.439092 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.440123 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.440427 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.440493 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.442704 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.442705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.442706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.442712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.444165 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.444530 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.444531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.444532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.444586 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.444790 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.444791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.444791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.444937 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.445012 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.445160 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.445231 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:36:55.447029 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc: Bootstrap starting.
I20260926 09:36:55.447793 23038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:55.448081 23038 log.cc:824] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:55.448973 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc: No bootstrap required, opened a new log
I20260926 09:36:55.450544 23038 raft_consensus.cc:348] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } }
I20260926 09:36:55.450717 23038 raft_consensus.cc:729] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1857b7f696428dad9fb61f73355abc, State: Initialized, Role: FOLLOWER
I20260926 09:36:55.450872 23038 consensus_queue.cc:260] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } }
I20260926 09:36:55.451143 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.451200 23038 sys_catalog.cc:564] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:55.451237 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:55.451216 23012 raft_consensus.cc:482] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 09:36:55.451404 23012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:55.452131 23012 raft_consensus.cc:504] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } }
I20260926 09:36:55.452384 23012 leader_election.cc:288] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [CANDIDATE]: Term 1 election: Requested vote from peers 69803b893503470f93b2e825c72d2f40 (127.0.98.61:36921), 6fe16528a4b948728dda65acfcea87cc (127.0.98.60:40845)
W20260926 09:36:55.453208 23219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:36:55.453264 23219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:36:55.453931 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af1857b7f696428dad9fb61f73355abc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6fe16528a4b948728dda65acfcea87cc"
I20260926 09:36:55.454070 23162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:55.455049 23162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af1857b7f696428dad9fb61f73355abc in term 1.
I20260926 09:36:55.455420 22987 leader_election.cc:302] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fe16528a4b948728dda65acfcea87cc, af1857b7f696428dad9fb61f73355abc; no voters:
I20260926 09:36:55.455531 23208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:36:55.455647 23208 raft_consensus.cc:686] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [term 1 LEADER]: Becoming Leader. State: Replica: af1857b7f696428dad9fb61f73355abc, State: Running, Role: LEADER
I20260926 09:36:55.455718 23208 consensus_queue.cc:237] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } }
I20260926 09:36:55.453651 22970 master_runner.cc:186] Error getting master registration for 127.0.98.61:36921: OK, instance_id { permanent_uuid: "69803b893503470f93b2e825c72d2f40" instance_seqno: 1790415415236126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:36:55.455860 22970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:36:55.456117 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1857b7f696428dad9fb61f73355abc. Latest consensus state: current_term: 1 leader_uuid: "af1857b7f696428dad9fb61f73355abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.456197 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:55.456636 23230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:36:55.456826 23230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:36:55.458455 23162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [term 1 FOLLOWER]: Refusing update from remote peer af1857b7f696428dad9fb61f73355abc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:55.458851 23208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:36:55.460235 23194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1857b7f696428dad9fb61f73355abc. Latest consensus state: current_term: 1 leader_uuid: "af1857b7f696428dad9fb61f73355abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.460319 23194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:55.462401 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af1857b7f696428dad9fb61f73355abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.462483 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:55.462845 23231 mvcc.cc:205] Tried to move back new op lower bound from 7333541541715402752 to 7333541541707296768. Current Snapshot: MvccSnapshot[applied={T|T < 7333541541715402752}]
I20260926 09:36:55.463200 23194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af1857b7f696428dad9fb61f73355abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.463280 23194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:55.463342 23232 mvcc.cc:205] Tried to move back new op lower bound from 7333541541715402752 to 7333541541707296768. Current Snapshot: MvccSnapshot[applied={T|T < 7333541541715402752}]
I20260926 09:36:55.470625 23113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } attempt: 3
I20260926 09:36:55.471747 23230 catalog_manager.cc:1347] Generated new cluster ID: a9d758942a434147bc28e86c65a94078
I20260926 09:36:55.471803 23230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:36:55.472601 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.454083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.98.61:36921)
0926 09:36:55.454137 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.454139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.459642 (+ 5503us) server_negotiation.cc:407] Connection header received
0926 09:36:55.459881 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.459882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.459892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.459920 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.462178 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.462814 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.465731 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.466041 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.466083 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.468113 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.468114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.468115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.468122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.469678 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.471763 (+ 2085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.471764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.471764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.471825 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.472033 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.472034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.472034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.472153 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.472248 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.472432 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.472511 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:36:55.475015 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.453892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.0.98.62:45205)
0926 09:36:55.453930 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.453932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.459538 (+ 5606us) server_negotiation.cc:407] Connection header received
0926 09:36:55.459783 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.459784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.459794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.459824 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.463917 (+ 4093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.464606 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.466391 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.467045 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.467091 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.470747 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.470748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.470748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.470755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.473768 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.474198 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.474199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.474200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.474256 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.474454 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.474455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.474455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.474572 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.474664 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.474855 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.474928 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:36:55.475442 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.454438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.0.98.60:40845)
0926 09:36:55.454471 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.454473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.461374 (+ 6901us) server_negotiation.cc:407] Connection header received
0926 09:36:55.461462 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.461463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.461472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.461499 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.463969 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.464683 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.466503 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.466790 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.466829 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.470701 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.470702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.470702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.470708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.473273 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.474436 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.474438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.474438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.474491 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.474741 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.474742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.474742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.474848 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.474923 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.475254 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.475358 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 09:36:55 dist-test-slave-21h8 krb5kdc[22957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:36:55.482573 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.471051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40845 (local address 127.0.0.1:42920)
0926 09:36:55.471139 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 09:36:55.471148 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:36:55.471193 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:36:55.475640 (+ 4447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:36:55.475641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:36:55.475772 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:36:55.475807 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.475808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.476639 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.476639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.477382 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:36:55.477384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.477850 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.477851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:36:55.477921 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.478331 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:36:55.478347 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:36:55.479699 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:36:55.481477 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.481478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.481478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.481669 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.481937 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:36:55.481937 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:36:55.481937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:36:55.481986 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:36:55.482313 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:36:55.482314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:36:55.482361 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:36:55.482411 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:36:55.482488 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 09:36:55.483035 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:55.471256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.0.98.60:40845)
0926 09:36:55.475505 (+ 4249us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.475506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.475512 (+ 6us) server_negotiation.cc:407] Connection header received
0926 09:36:55.475527 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.475528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.475536 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.475579 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.475900 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.476554 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.477470 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:55.477752 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:55.477797 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:55.479794 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:36:55.479795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:36:55.479796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:36:55.479802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:36:55.481382 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.481748 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.481749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.481749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.481824 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:36:55.482051 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:36:55.482052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:36:55.482052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:36:55.482211 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:36:55.482720 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:55.482787 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 09:36:55.482907 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4237,"thread_start_us":109,"threads_started":1}
I20260926 09:36:55.484350 23113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40: Bootstrap starting.
I20260926 09:36:55.485004 23113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40: Neither blocks nor log segments found. Creating new log.
I20260926 09:36:55.485261 23113 log.cc:824] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40: Log is configured to *not* fsync() on all Append() calls
I20260926 09:36:55.486878 23113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40: No bootstrap required, opened a new log
I20260926 09:36:55.488657 23113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } }
I20260926 09:36:55.488787 23113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69803b893503470f93b2e825c72d2f40, State: Initialized, Role: FOLLOWER
I20260926 09:36:55.488864 23113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } }
I20260926 09:36:55.489141 23113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:36:55.490266 23087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af1857b7f696428dad9fb61f73355abc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "69803b893503470f93b2e825c72d2f40"
I20260926 09:36:55.490381 23087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:36:55.491123 23087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af1857b7f696428dad9fb61f73355abc in term 1.
I20260926 09:36:55.491297 23086 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [term 1 FOLLOWER]: Refusing update from remote peer af1857b7f696428dad9fb61f73355abc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:36:55.492337 23228 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [LEADER]: Connected to new peer: Peer: permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:36:55.494562 23249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:36:55.494624 23249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:36:55.494932 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af1857b7f696428dad9fb61f73355abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.495782 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:55.495868 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1857b7f696428dad9fb61f73355abc. Latest consensus state: current_term: 1 leader_uuid: "af1857b7f696428dad9fb61f73355abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.495934 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:55.495972 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af1857b7f696428dad9fb61f73355abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.496016 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:36:55.499729 23251 mvcc.cc:205] Tried to move back new op lower bound from 7333541541715402752 to 7333541541707296768. Current Snapshot: MvccSnapshot[applied={T|T < 7333541541715402752}]
I20260926 09:36:55.502947 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af1857b7f696428dad9fb61f73355abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1857b7f696428dad9fb61f73355abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45205 } } peers { permanent_uuid: "69803b893503470f93b2e825c72d2f40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36921 } } peers { permanent_uuid: "6fe16528a4b948728dda65acfcea87cc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40845 } } }
I20260926 09:36:55.503046 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc [sys.catalog]: This master's current role is: LEADER
I20260926 09:36:55.504022 23230 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:36:55.504742 23230 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:36:55.512198 23230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af1857b7f696428dad9fb61f73355abc: Generated new TSK 0
I20260926 09:36:55.512404 23230 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:36:55.535867 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.534372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.0.98.61:36921)
0926 09:36:55.534418 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.534420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.535200 (+ 780us) server_negotiation.cc:407] Connection header received
0926 09:36:55.535305 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.535306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.535334 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.535375 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.535798 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50928: BlockingRecv error: recv got EOF from 127.0.0.1:50928 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 09:36:55.536586 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.534714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.0.98.60:40845)
0926 09:36:55.534753 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.534755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.536090 (+ 1335us) server_negotiation.cc:407] Connection header received
0926 09:36:55.536186 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.536187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.536197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.536225 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.536529 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42930: BlockingRecv error: recv got EOF from 127.0.0.1:42930 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 09:36:55.537257 23041 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.535036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.0.98.62:45205)
0926 09:36:55.535084 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.535086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.536793 (+ 1707us) server_negotiation.cc:407] Connection header received
0926 09:36:55.536885 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.536886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.536896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.536925 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.537201 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48738: BlockingRecv error: recv got EOF from 127.0.0.1:48738 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 09:36:55.540329 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.539574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.0.98.61:36921)
0926 09:36:55.539606 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.539608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.539822 (+ 214us) server_negotiation.cc:407] Connection header received
0926 09:36:55.539923 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.539924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.539941 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.539969 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.540287 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50938: BlockingRecv error: recv got EOF from 127.0.0.1:50938 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:36:55.543252 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.542531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.0.98.61:36921)
0926 09:36:55.542561 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.542563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.542790 (+ 227us) server_negotiation.cc:407] Connection header received
0926 09:36:55.542876 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.542877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.542887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.542914 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.543200 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50940: BlockingRecv error: recv got EOF from 127.0.0.1:50940 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:36:55.547053 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.546374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.0.98.61:36921)
0926 09:36:55.546406 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.546407 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.546602 (+ 195us) server_negotiation.cc:407] Connection header received
0926 09:36:55.546689 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.546690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.546699 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.546727 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.547013 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50946: BlockingRecv error: recv got EOF from 127.0.0.1:50946 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:36:55.552868 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.552075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.0.98.61:36921)
0926 09:36:55.552125 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.552127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.552375 (+ 248us) server_negotiation.cc:407] Connection header received
0926 09:36:55.552480 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.552481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.552490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.552518 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.552827 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50948: BlockingRecv error: recv got EOF from 127.0.0.1:50948 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 09:36:55.562849 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.561965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.0.98.61:36921)
0926 09:36:55.562008 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.562010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.562227 (+ 217us) server_negotiation.cc:407] Connection header received
0926 09:36:55.562323 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.562324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.562333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.562375 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.562792 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50950: BlockingRecv error: recv got EOF from 127.0.0.1:50950 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:36:55.580963 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.580112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.0.98.61:36921)
0926 09:36:55.580149 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.580151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.580418 (+ 267us) server_negotiation.cc:407] Connection header received
0926 09:36:55.580520 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.580521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.580531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.580560 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.580912 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50958: BlockingRecv error: recv got EOF from 127.0.0.1:50958 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 09:36:55.615236 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.614418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.0.98.61:36921)
0926 09:36:55.614466 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.614469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.614584 (+ 115us) server_negotiation.cc:407] Connection header received
0926 09:36:55.614652 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.614653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.614667 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.614751 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.615159 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50974: BlockingRecv error: recv got EOF from 127.0.0.1:50974 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 09:36:55.681217 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.680471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.0.98.61:36921)
0926 09:36:55.680551 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.680553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.680642 (+ 89us) server_negotiation.cc:407] Connection header received
0926 09:36:55.680739 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.680740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.680752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.680825 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.681155 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50988: BlockingRecv error: recv got EOF from 127.0.0.1:50988 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 09:36:55.811203 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:55.810412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.0.98.61:36921)
0926 09:36:55.810507 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:55.810510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:55.810557 (+ 47us) server_negotiation.cc:407] Connection header received
0926 09:36:55.810665 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:55.810666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:55.810678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:55.810732 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:55.811123 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50998: BlockingRecv error: recv got EOF from 127.0.0.1:50998 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 09:36:56.068846 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:36:56.068278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.0.98.61:36921)
0926 09:36:56.068367 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:56.068369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:56.068381 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:36:56.068410 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:56.068411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:56.068422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:56.068492 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:36:56.068796 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51004: BlockingRecv error: recv got EOF from 127.0.0.1:51004 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 09:36:56.419632 23205 catalog_manager.cc:1259] Loaded cluster ID: a9d758942a434147bc28e86c65a94078
I20260926 09:36:56.419723 23205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc: loading cluster ID for follower catalog manager: success
I20260926 09:36:56.420570 23205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc: acquiring CA information for follower catalog manager: success
I20260926 09:36:56.420876 23205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fe16528a4b948728dda65acfcea87cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:36:56.495018 23249 catalog_manager.cc:1259] Loaded cluster ID: a9d758942a434147bc28e86c65a94078
I20260926 09:36:56.495082 23249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40: loading cluster ID for follower catalog manager: success
I20260926 09:36:56.495975 23249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40: acquiring CA information for follower catalog manager: success
I20260926 09:36:56.496299 23249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69803b893503470f93b2e825c72d2f40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:36:56.584463 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:56.581998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.0.98.61:36921)
0926 09:36:56.582126 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:56.582129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:56.582146 (+ 17us) server_negotiation.cc:407] Connection header received
0926 09:36:56.582230 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:56.582231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:56.582281 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:56.582371 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:56.582516 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.583036 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.583773 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.583976 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.584031 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:56.584175 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:56.584281 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:56.584340 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:56.584380 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:36:56.584398 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260926 09:36:56.588774 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:56.586253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.0.98.60:40845)
0926 09:36:56.586400 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:56.586402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:56.586412 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:36:56.586456 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:56.586457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:56.586509 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:56.586617 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:56.586735 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.587214 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.587946 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.588162 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.588239 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:56.588412 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:56.588535 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:56.588615 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:56.588668 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 09:36:56.588687 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260926 09:36:56.593225 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:56.590339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.0.98.61:36921)
0926 09:36:56.590386 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:56.590387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:56.590713 (+ 326us) server_negotiation.cc:407] Connection header received
0926 09:36:56.590772 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:56.590773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:56.590785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:56.590837 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:56.591006 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.591563 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.592373 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.592630 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.592709 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:56.592885 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:56.592915 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:56.592965 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:56.593130 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 09:36:56.593156 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:36:56.593469 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:56.590545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.0.98.60:40845)
0926 09:36:56.590592 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:56.590593 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:56.590870 (+ 277us) server_negotiation.cc:407] Connection header received
0926 09:36:56.590897 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:56.590898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:56.590907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:56.590962 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:56.591068 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.591743 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.592715 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.592986 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.593049 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:56.593177 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:56.593206 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:56.593235 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:56.593351 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 09:36:56.593378 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:36:56.593582 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:36:56.590257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.0.98.62:45205)
0926 09:36:56.590415 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 09:36:56.590417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:36:56.590425 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:36:56.590473 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:36:56.590474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:36:56.590507 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:36:56.590602 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:36:56.590707 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.591508 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.592547 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:36:56.592890 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:36:56.592954 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:36:56.593113 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:36:56.593215 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:36:56.593272 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:36:56.593483 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 09:36:56.593506 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":48,"threads_started":1}
I20260926 09:36:56.604291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 22970
I20260926 09:36:56.609452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 23045
I20260926 09:36:56.614352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TYWI0/build/release/bin/kudu with pid 23120
2026-09-26T09:36:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2802 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5574 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270509 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST